Sensor modification based on an annotated environmental model
Systems, methods, and computer-readable storage mediums which use annotated environmental models for sensor modification are disclosed. A computing system receives an environmental model for a vehicle. The environmental model can include data from a plurality of modalities. Object annotations are received from sensors which are used to modify the environmental model, creating an annotated environmental model. A classification of a current situation is generated using the annotated environmental model, and a sensor is modified using the classification.
1. A method comprising:
receiving, at a processor of a sensor fusion system of a vehicle, an environmental model comprising raw measurement data from a plurality of sensors of the vehicle, the plurality of sensors having distinct modalities, wherein the raw measurement data is temporally aligned in the environmental model and spatially aligned in the environmental model based on where the sensors are mounted in the vehicle;
receiving, at the processor, object annotations indicating a detection of at least one object based on the raw measurement data in the environmental model, the object annotations comprising a confidence level associated with the detection of the at least one object;
modifying, via the processor, the environmental model to include the object annotations, resulting in an annotated environmental model, wherein the object annotations are temporally aligned and spatially aligned in the annotated environmental model;
generating, via the processor, a classification result of a current situation of the vehicle based, at least in part, on the raw measurement data and the object annotations in the annotated environmental model; and
transmitting, via the processor, the classification result to a processor associated with driving functionality of the vehicle, wherein the processor associated with driving functionality is configured to control operation the vehicle based, at least in part, on the classification result.
2. The method of claim 1 , further comprising:
generating, via the processor and based on the classification result, a feedback signal; and
transmitting, from the processor to at least one sensor in the plurality of sensors, the feedback signal which causes at least one of repositioning the at least one sensor, or expanding a field of view of the at least one sensor, or changing a rate of exposure of the at least one sensor.
3. The method of claim 1 , wherein the classification result is further determined based on one or more of a region of interest in the environmental model, a predicted environmental model, and localization information.
4. The method of claim 1 , wherein the distinct modalities comprise one or more of raw measurement data from one or more sensors on the vehicle, fused sensor data, high level data, and data from one or more external sources.
5. The method of claim 1 , further comprising providing the classification result to a processor configured to adjust a region of interest in the environmental model based, at least in part, on the classification result.
6. The method of claim 1 , further comprising:
determining that the classification result identifies an uncertain vehicle situation; and
providing classification data associated with the classification result to an external computing system located away from the vehicle, wherein the external computing system is configured to use the classification data to generate an update to the sensor fusion system based on the uncertain vehicle situation.
7. The method of claim 6 , further comprising:
receiving, from the external computing system, the update; and
modifying the processor based on the update from the external computing system.
8. The method of claim 1 , further comprising:
storing the annotated environmental model and classification result to a memory associated with the vehicle; and
upon detecting an incident associated with the vehicle, evaluating the incident using the annotated environmental model and classification result stored in the memory.
9. An apparatus comprising at least one memory device storing instructions configured to cause one or more processing devices of a sensor fusion system to perform operations comprising:
receiving an environmental model comprising raw measurement data from a plurality of sensors of the vehicle, the plurality of sensors having distinct modalities, wherein the raw measurement data is temporally aligned in the environmental model and spatially aligned in the environmental model based on where the sensors are mounted in the vehicle;
receiving object annotations indicating a detection of at least one object based on the raw measurement data in the environmental model, the object annotations comprising a confidence level associated with the detection of the at least one object;
modifying the environmental model to include the object annotations, resulting in an annotated environmental model, wherein the object annotations are temporally aligned and spatially aligned in the annotated environmental model;
generating a classification result of a current situation of the vehicle based, at least in part, on the raw measurement data and the object annotations in the annotated environmental model; and
transmitting the classification result to a processor associated with driving functionality of the vehicle, wherein the processor associated with driving functionality is configured to control operation the vehicle based, at least in part, on the classification result.
10. The apparatus of claim 9 , wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising:
generating, based on the classification result, a feedback signal; and
transmitting, to the at least one sensor in the plurality of sensors, the feedback signal, causing at least one of repositioning the at least one sensor, or expanding a field of view of the at least one sensor, or changing a rate of exposure of the at least one sensor.
11. The apparatus of claim 9 , wherein the classification result is further determined based on one or more of a region of interest in the environmental model, a predicted environmental model, and localization information.
12. The apparatus of claim 9 , wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising providing the classification result to a processor configured to adjust a region of interest in the environmental model based, at least in part, on the classification result.
13. The apparatus of claim 9 , wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising:
determining that the classification result identifies an uncertain vehicle situation; and
providing classification data associated with the classification result to an external computing system located away from the vehicle, wherein the external computing system is configured to use the classification data to generate an update to the sensor fusion system based on the uncertain vehicle situation.
14. The apparatus of claim 13 , wherein the instructions are further configured to cause the one or more processing devices to perform operations comprising:
receiving, from the external computing system, the update; and
modifying the one or more processing devices based on the update from the external computing system.
15. A sensor fusion system comprising:
a memory device configured to store machine-readable instructions; and
one or more processing devices winch, in response to executing the machine-readable, instructions, perform operations comprising:
receiving an environmental model comprising raw measurement data from a plurality of sensors of the vehicle, the plurality of sensors having distinct modalities, wherein the raw measurement data is temporally aligned in the environmental model and spatially aligned in the environmental model based on where the sensors are mounted in the vehicle;
receiving object annotations indicating a detection of at least one object based on the raw measurement data in the environmental model, the object annotations comprising a confidence level associated with the detection of the at least one object;
modifying the environment model to include the object annotations, resulting in an annotated environmental model, wherein the object annotations are temporally aligned and spatially aligned in the annotated environmental model;
generating a classification result of a current situation of the vehicle based, at least in part, on the raw measurement data and the object annotations in the annotated environmental model; and
transmitting the classification result to a processor associated with driving functionality of the vehicle, wherein the processor associated with driving functionality is configured to control operation the vehicle based, at least in part, on the classification result.
16. The sensor fusion system of claim 15 , wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:
generate, based on the classification result, a feedback signal; and
transmit, to the at least one sensor in the plurality of sensors, the feedback signal, causing at least one of repositioning the at least one sensor, or expanding a field of view of at least one sensor, or changing a rate of exposure of the at least one sensor.
17. The sensor fusion system of claim 15 , wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to provide the classification result to a processor configured to adjust a region of interest in the environmental model based, at least in part, on the classification result.
18. The sensor fusion system of claim 15 , wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:
determine that the classification result identifies an uncertain vehicle situation; and
provide classification data associated with the classification result to an external computing system located away from the vehicle, wherein the external computing system is configured to use the classification data to generate an update to the sensor fusion system based on the uncertain vehicle situation.
19. The sensor fusion system of claim 18 , wherein the one or more processing devices, in response to executing the machine-readable instructions, are configured to:
receive, from the external computing system, the update; and
modify the one or more processing devices based on the update from the external computing system.