Perception anomaly detection for autonomous driving
A unified framework for detecting perception anomalies in autonomous driving systems is described. The perception anomaly detection framework takes an input image from a camera in or on a vehicle and identifies anomalies as belonging to one of three categories. Lens anomalies are associated with poor sensor conditions, such as water, dirt, or overexposure. Environment anomalies are associated with unfamiliar changes to an environment. Finally, object anomalies are associated with unknown objects. After perception anomalies are detected, the results are sent downstream to cause a behavior change of the vehicle.
1 . A computer-implemented method comprising:
receiving, by a computer in a vehicle, an image obtained by a camera of the vehicle;
determining a set of features of the image;
identifying, based on the set of features, an anomaly associated with the camera;
mapping, based on the set of features, a plurality of uncertainty values to the image; and
determining, based on the anomaly and the plurality of uncertainty values, a status of the camera,
wherein the image includes an environment external to the vehicle, the method further comprising:
masking at least a portion of the set of features;
generating a reconstructed set of features based on the masked portion; and
based on a comparison between the set of features and the reconstructed set of features, identifying an anomaly associated with the environment.
2 . The computer-implemented method of claim 1 , further comprising:
causing the vehicle to perform a driving operation based on the status of the camera; or
engaging a redundant camera based on the status of the camera.
3 . The computer-implemented method of claim 1 , wherein the anomaly associated with the camera includes at least one of: water, mud, dirt, or camera exposure.
4 . The computer-implemented method of claim 1 , wherein the plurality of uncertainty values are respectively mapped to each pixel of the image.
5 . The computer-implemented method of claim 1 , wherein determining the status of the camera includes:
determining, based on the plurality of uncertainty values, a number of pixels that have an uncertainty value greater than a threshold uncertainty value; and
determining the status of the camera based on the number of pixels.
6 . The computer-implemented method of claim 1 , wherein the comparison is performed by:
computing a difference between a first map including the set of features and a second map including the reconstructed set of features; and
producing an anomaly map based on the difference.
7 . The computer-implemented method of claim 6 , further comprising:
identifying a foreground object in the anomaly map; and
subtracting the foreground object from the anomaly map.
8 . The computer-implemented method of claim 1 , further comprising:
identifying an object in the image; and
classifying the object as an anomaly based on a determination that the object is outside a set of predefined categories.
9 . A system for autonomous driving comprising:
a camera coupled to a vehicle;
a processor; and
a memory storing instructions, execution of which by the processor causes the system to:
obtain, by the camera, an image;
determine a set of features of the image;
identify, based on the set of features, an anomaly associated with the camera;
map, based on the set of features, a plurality of uncertainty values to the image; and
determine, based on the anomaly and the plurality of uncertainty values, a status of the camera,
wherein the image includes an environment external to the vehicle, the system further caused to:
mask at least a portion of the set of features;
generate a reconstructed set of features based on the masked portion; and
based on a comparison between the set of features and the reconstructed set of features, identify an anomaly associated with the environment.
10 . The system of claim 9 , further caused to:
causing the vehicle to perform a driving operation based on the status of the camera; or
engaging a redundant camera based on the status of the camera.
11 . The system of claim 9 , wherein the anomaly associated with the camera includes at least one of: water, mud, dirt, or camera exposure.
12 . The system of claim 9 , wherein the status of the camera is determined by:
determining, based on the plurality of uncertainty values, a number of pixels that have an uncertainty value greater than a threshold uncertainty value; and
determining the status of the camera based on the number of pixels.
13 . The system of claim 9 , wherein the comparison is performed by:
computing a difference between a first map including the set of features and a second map including the reconstructed set of features; and
producing an anomaly map based on the difference.
14 . A non-transitory computer-readable medium storing instructions, execution of which by a processor of a computing system causes the computing system to:
receive an image obtained by a camera of a vehicle;
determine a set of features of the image;
identify, based on the set of features, a first anomaly associated with the camera;
mask at least a portion of the set of features;
generate a reconstructed set of features based on the masked portion;
based on a comparison between the set of features and the reconstructed set of features, identify a second anomaly associated with an environment external to the vehicle;
identify a third anomaly associated with an object in the image based on a determination that the object is outside a set of predefined categories; and
causing the vehicle to perform a driving operation based on the first anomaly, the second anomaly, or the third anomaly.
15 . The non-transitory computer-readable medium of claim 14 , the computing system further caused to:
map, based on the set of features, a plurality of uncertainty values to the image; and
determine, based on the first anomaly and the plurality of uncertainty values, a status of the camera.
16 . The non-transitory computer-readable medium of claim 14 , wherein the comparison is performed by:
computing a difference between a first map including the set of features and a second map including the reconstructed set of features; and
producing an anomaly map based on the difference.
17 . The non-transitory computer-readable medium of claim 16 , the computing system further caused to:
identify a foreground object in the anomaly map; and
subtract the foreground object from the anomaly map.
18 . The non-transitory computer-readable medium of claim 16 , the computing system further caused to:
map, based on the set of features, a plurality of uncertainty values to the image; and
produce an occupancy grid based on the plurality of uncertainty values and the anomaly map.
19 . The system of claim 9 , wherein the plurality of uncertainty values are respectively mapped to each pixel of the image.
20 . The system of claim 9 , wherein the processor further causes the system to:
identify an object in the image; and
classify the object as an anomaly based on a determination that the object is outside a set of predefined categories.