Detecting and responding to vehicle sensor data inaccuracies
Sensor data captured by multiple vehicle sensors may be analyzed and compared to detect sensor attacks and/or other sensor anomalies. Vehicle systems described herein may determine overlapping regions in the fields of view of multiple sensors of a vehicle, including sensors of the same or different modalities. The sensor outputs from the overlapping regions may be compared to determine when a sensor output is unexpected based on the outputs of overlapping sensors. When an unexpected sensor output is detected, the vehicle systems may analyze the sensor data to determine whether the cause of the unexpected sensor output is benign or a potential malicious attack on a sensor or group of sensors. The vehicle systems also may determine and perform responses of the vehicle to various sensor anomalies, including controlling the vehicle to avoid potential threats and/or capturing additional sensor data relating to the causes of the sensor anomalies.
1 . A vehicle comprising:
a plurality of sensors configured to capture sensor data of an environment external to the vehicle, the plurality of sensors including a first sensor having a first field of view relative to the vehicle and a second sensor having a second field of view relative to the vehicle, wherein the first field of view at least partially overlaps with the second field of view;
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the vehicle to perform operations comprising:
receiving first sensor data from the first sensor having the first field of view relative to the vehicle;
receiving second sensor data from the second sensor having the second field of view relative to the vehicle, wherein the second field of view comprises an overlapping region between the first field of view and the second field of view;
determining an inaccuracy in the first sensor data, based at least in part on an inconsistency between a first portion of the first sensor data within the overlapping region and a second portion of the second sensor data within the overlapping region;
determining an intent of an agent in the environment to interfere with operation of the first sensor;
determining, based at least in part on the second sensor data, a cause associated with the inaccuracy in the first sensor data, wherein determining the cause comprises determining that the inaccuracy is associated with an action of the agent and the intent of the agent in the environment to interfere with the operation of the first sensor, wherein a position associated with the agent is within at least one of the first field of view or the second field of view; and
controlling operation of the vehicle based at least in part on the cause associated with the inaccuracy, wherein controlling operation of the vehicle comprises an action of the vehicle, where the action of the vehicle is based at least in part on the intent of the agent in the environment to interfere with the operation of the first sensor.
2 . The vehicle of claim 1 , wherein determining the cause associated with the inaccuracy includes at least one of:
determining an emission from an object associated with the agent, the emission causing a degradation of the first sensor data; or
determining an object secured to the vehicle by the agent, the object modifying at least a portion of the first field of view of the first sensor.
3 . The vehicle of claim 1 , wherein determining the cause associated with the inaccuracy comprises at least one of:
determining a time duration associated with the inaccuracy; or
performing a remedial operation on the first sensor, based at least in part on the inaccuracy.
4 . The vehicle of claim 1 , wherein controlling operation of the vehicle comprises:
determining a location associated with the agent in the environment; and
controlling a sensor of the vehicle to capture additional data of the location.
5 . The vehicle of claim 1 , wherein controlling operation of the vehicle comprises:
determining a first control trajectory for controlling the vehicle within the environment, wherein the first control trajectory was selected from a set of potential trajectories for controlling the vehicle before determining the cause associated with the inaccuracy; and
based at least in part on determining that the inaccuracy is associated with the action of the agent, determining a different second control trajectory from the set of potential trajectories, for controlling the vehicle.
6 . A method comprising:
receiving first sensor data from a first sensor of a vehicle in an environment, the first sensor having a first field of view relative to the vehicle;
receiving second sensor data from a second sensor of the vehicle, the second sensor having a second field of view relative to the vehicle, wherein the second field of view comprises an overlapping region between the first field of view and the second field of view;
determining an inaccuracy in the first sensor data, based at least in part on an inconsistency between a first portion of the first sensor data within the overlapping region and a second portion of the second sensor data within the overlapping region;
determining an intent of an agent in the environment to interfere with operation of the first sensor;
determining a cause associated with the inaccuracy, wherein determining the cause comprises determining that the inaccuracy is associated with an action of the agent and the intent of the agent in the environment to interfere with the operation of the first sensor, wherein a position associated with the agent is within at least one of the first field of view or the second field of view; and
controlling operation of the vehicle based at least in part on the cause associated with the inaccuracy in the first sensor data, wherein controlling operation of the vehicle comprises an action of the vehicle, where the action of the vehicle is based at least in part on the intent of the agent in the environment to interfere with the operation of the first sensor.
7 . The method of claim 6 , wherein determining the cause associated with the inaccuracy includes at least one of:
determining an emission from an object associated with the agent, the emission causing a degradation of the first sensor data; or
determining an object secured to the vehicle by the agent, the object modifying at least a portion of the first field of view of the first sensor.
8 . The method of claim 6 , wherein determining the cause associated with the inaccuracy comprises at least one of:
determining a time duration associated with the inaccuracy; or
performing a remedial operation on the first sensor, based at least in part on the inaccuracy.
9 . The method of claim 6 , wherein the inaccuracy in the first sensor data includes at least one of:
a false positive object detection; or
a false negative object detection.
10 . The method of claim 6 , wherein
the first sensor is configured to capture data of a first sensor modality, and
the second sensor is configured to capture data of a second sensor modality different from the first sensor modality.
11 . The method of claim 6 , wherein controlling operation of the vehicle comprises:
determining a location associated with the agent in the environment; and
controlling a sensor of the vehicle to capture additional data of the location.
12 . The method of claim 6 , wherein controlling operation of the vehicle comprises:
determining a first control trajectory for controlling the vehicle within the environment, wherein the first control trajectory was selected from a set of potential trajectories for controlling the vehicle before determining the cause associated with the inaccuracy; and
based at least in part on determining that the inaccuracy is associated with the action of the agent, determining a different second control trajectory from the set of potential trajectories, for controlling the vehicle.
13 . The method of claim 6 , wherein controlling operation of the vehicle comprises:
determining a location associated with the agent in the environment; and
determining a driving route for the vehicle, based at least in part on the location.
14 . One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving first sensor data from a first sensor of a vehicle in an environment, the first sensor having a first field of view relative to the vehicle;
receiving second sensor data from a second sensor of the vehicle, the second sensor having a second field of view relative to the vehicle, wherein the second field of view comprises an overlapping region between the first field of view and the second field of view;
determining an inaccuracy in the first sensor data, based at least in part on an inconsistency between a first portion of the first sensor data within the overlapping region and a second portion of the second sensor data within the overlapping region;
determining an intent of an agent in the environment to interfere with operation of the first sensor;
determining a cause associated with the inaccuracy, wherein determining the cause comprises determining that the inaccuracy is associated with an action of the agent and the intent of the agent in the environment to interfere with the operation of the first sensor, wherein a position associated with the agent is within at least one of the first field of view or the second field of view; and
controlling operation of the vehicle based at least in part on the cause associated with the inaccuracy in the first sensor data, wherein controlling operation of the vehicle comprises an action of the vehicle, where the action of the vehicle is based at least in part on the intent of the agent in the environment to interfere with the operation of the first sensor.
15 . The one or more non-transitory computer-readable media of claim 14 , wherein determining the cause associated with the inaccuracy includes at least one of:
determining an emission from an object associated with the agent, the emission causing a degradation of the first sensor data; or
determining an object secured to the vehicle by the agent, the object modifying at least a portion of the first field of view of the first sensor.
16 . The one or more non-transitory computer-readable media of claim 14 , wherein determining the cause associated with the inaccuracy comprises at least one of:
determining a time duration associated with the inaccuracy; or
performing a remedial operation on the first sensor, based at least in part on the inaccuracy.
17 . The one or more non-transitory computer-readable media of claim 14 , wherein the inaccuracy in the first sensor data includes at least one of:
a false positive object detection; or
a false negative object detection.
18 . The one or more non-transitory computer-readable media of claim 14 , wherein:
the first sensor is configured to capture data of a first sensor modality, and
the second sensor is configured to capture data of a second sensor modality different from the first sensor modality.
19 . The one or more non-transitory computer-readable media of claim 14 , wherein controlling operation of the vehicle comprises:
determining a location associated with the agent in the environment; and
controlling a sensor of the vehicle to capture additional data of the location.
20 . The one or more non-transitory computer-readable media of claim 14 , wherein controlling operation of the vehicle comprises:
determining a first control trajectory for controlling the vehicle within the environment, wherein the first control trajectory was selected from a set of potential trajectories for controlling the vehicle before determining the cause associated with the inaccuracy; and
based at least in part on determining that the inaccuracy is associated with the action of the agent, determining a different second control trajectory from the set of potential trajectories, for controlling the vehicle.