Combining rule-based and learned sensor fusion for autonomous systems and applications
In various examples, systems and methods are disclosed that perform sensor fusion using rule-based and learned processing methods to take advantage of the accuracy of learned approaches and the decomposition benefits of rule-based approaches for satisfying higher levels of safety requirements. For example, in-parallel and/or in-serial combinations of early rule-based sensor fusion, late rule-based sensor fusion, early learned sensor fusion, or late learned sensor fusion may be used to solve various safety goals associated with various required safety levels at a high level of accuracy and precision. In embodiments, learned sensor fusion may be used to make more conservative decisions than the rule-based sensor fusion (as determined using, e.g., severity (S), exposure (E), and controllability (C) (SEC) associated with a current safety goal), but the rule-based sensor fusion may be relied upon where the learned sensor fusion decision may be less conservative than the corresponding rule-based sensor fusion.
1. One or more processors comprising:
processing circuitry to:
generate a first fused output by applying learned fusion to first data generated using a first sensor processing pipeline and second data generated using a second sensor processing pipeline;
generate a second fused output by applying rule-based fusion to the first data, the second data, and the first fused output; and
perform one or more operations based at least on the second fused output,
wherein the first fused output is capable of compliance with a first automotive integrity safety level (ASIL) and the second fused output is capable of compliance with a second ASIL level, the second ASIL level being a higher ASIL level than the first ASIL level.
2. The one or more processors of claim 1 , wherein:
the first sensor processing pipeline includes at least a first sensor and a first processing component that processes first sensor data obtained using the first sensor to compute a first intermediate output; and
the second sensor processing pipeline includes at least a second sensor and a second processing component that processes second sensor data obtained using the second sensor to compute a second intermediate output,
wherein at least one of the first data or the second data represents at least one of the first intermediate output or the second intermediate output.
3. The one or more processors of claim 2 , wherein the first processing component and the second processing component include at least one of a learned processing component or a rule-based processing component.
4. The one or more processors of claim 3 , wherein the learned processing component includes a deep neural network (DNN).
5. The one or more processors of claim 1 , wherein the first data and the second data correspond to sensor data obtained using at least one of the first sensor processing pipeline or the second sensor processing pipeline.
6. The one or more processors of claim 1 , wherein the first data and the second data correspond to one or more outputs of one or more processing components of at least one of the first sensor processing pipeline or the second sensor processing pipeline, the processing circuitry processing at least sensor data obtained using a respective sensor of the first sensor processing pipeline or the second sensor processing pipeline.
7. The one or more processors of claim 1 , wherein the learned fusion corresponds to at least one of early learned sensor fusion or late learned sensor fusion.
8. The one or more processors of claim 1 , wherein the first sensor processing pipeline and the second sensor processing pipeline are each capable of compliance with a lower ASIL level than the second ASIL level.
9. The one or more processors of claim 1 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
10. A sensor fusion architecture comprising:
a first sensor processing pipeline including at least a first sensor to obtain first sensor data and a first processing component to process the first sensor data to compute a first intermediate output;
a second sensor processing pipeline including at least a second sensor to obtain second sensor data and a second processing component to process the second sensor data to compute a second intermediate output;
a learned sensor fusion processing component to process data from the first sensor processing pipeline and the second sensor processing pipeline to generate a third intermediate output; and
a rule-based sensor fusion processing component to process the first intermediate output, the second intermediate output, and the third intermediate output to generate a fused output,
wherein the first intermediate output, the second intermediate output, and the third intermediate output correspond to one or more first safety levels lower than a second safety level corresponding to the fused output.
11. The sensor fusion architecture of claim 10 , wherein the data from the first sensor processing pipeline and the second sensor processing pipeline includes the first sensor data and the second sensor data.
12. The sensor fusion architecture of claim 10 , wherein the data from the first sensor processing pipeline and the second sensor processing pipeline includes data representative of the first intermediate output and the second intermediate output.
13. The sensor fusion architecture of claim 10 , wherein the one or more first safety levels correspond to one or more first safety integrity levels and the second safety level corresponds to a safety integrity level.
14. The sensor fusion architecture of claim 10 , wherein the first intermediate output and the second intermediate output further correspond to a first safety goal, the third intermediate output further corresponds to a second safety goal, and the fused output further corresponds to one of the first safety goal or the second safety goal.
15. The sensor fusion architecture of claim 10 , wherein the fused output is used to perform one or more operations of a machine.
16. A system comprising:
one or more processors to:
generate a first fused output by applying learned fusion to first data generated using a first sensor processing pipeline and second data generated using a second sensor processing pipeline;
generate a second fused output by applying rule-based fusion to the first data, the second data, and the first fused output; and
perform one or more operations based at least on the second fused output,
wherein the first fused output is capable of compliance with a first automotive integrity safety level (ASIL) and the second fused output is capable of compliance with a second ASIL level, the second ASIL level being a higher ASIL level than the first ASIL level.
17. The system of claim 16 , wherein:
the first sensor processing pipeline includes at least a first sensor and a first processing component that processes first sensor data obtained using the first sensor to compute a first intermediate output; and
the second sensor processing pipeline includes at least a second sensor and a second processing component that processes second sensor data obtained using the second sensor to compute a second intermediate output,
wherein at least one of the first data or the second data represents at least one of the first intermediate output or the second intermediate output.
18. The system of claim 17 , wherein the first processing component and the second processing component include at least one of a learned processing component or a rule-based processing component.
19. The system of claim 18 , wherein the learned processing component includes a deep neural network (DNN).
20. The system of claim 16 , wherein the first data and the second data correspond to sensor data obtained using at least one of the first sensor processing pipeline or the second sensor processing pipeline.
21. The system of claim 16 , wherein the first data and the second data correspond to one or more outputs of one or more processing components of at least one of the first sensor processing pipeline or the second sensor processing pipeline, the one or more processing components processing at least sensor data obtained using a respective sensor of the first sensor processing pipeline or the second sensor processing pipeline.
22. The system of claim 16 , wherein the learned fusion corresponds to at least one of early learned sensor fusion or late learned sensor fusion.
23. The system of claim 16 , wherein the first sensor processing pipeline and the second sensor processing pipeline are each in compliance with a lower ASIL level than the second ASIL level.
24. The system of claim 16 wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
25. A method comprising:
generating, based at least on a first sensor processing pipeline using a first processing component to process first sensor data obtained using a first sensor, a first intermediate output;
generating, based at least on a second sensor processing pipeline using a second processing component to process second sensor data obtained using a second sensor, a second intermediate output;
processing data from the first sensor processing pipeline and the second sensor processing pipeline to generate a third intermediate output; and
processing the first intermediate output, the second intermediate output, and the third intermediate output to generate a fused output,
wherein the first intermediate output, the second intermediate output, and the third intermediate output correspond to one or more first safety levels lower than a second safety level corresponding to the fused output.
26. The method of claim 25 , wherein the data from the first sensor processing pipeline and the second sensor processing pipeline includes the first sensor data and the second sensor data.
27. The method of claim 25 , wherein the data from the first sensor processing pipeline and the second sensor processing pipeline includes data representative of the first intermediate output and the second intermediate output.
28. The method of claim 25 , wherein the one or more first safety levels correspond to one or more first safety integrity levels and the second safety level corresponds to a second safety integrity level.
29. The method of claim 25 , wherein the first intermediate output and the second intermediate output correspond to a first safety goal, the third intermediate output corresponds to a second safety goal, and the fused output corresponds to one of the first safety goal or the second safety goal.
30. The sensor fusion architecture of claim 10 , wherein the first sensor data and the second sensor data also correspond to the first ASIL.
31. The method of claim 25 , wherein the processing the data from the first sensor processing pipeline and the second sensor processing pipeline uses a learned sensor fusion component.
32. The method of claim 25 , wherein the processing the first intermediate output, the second intermediate output, and the third intermediate output uses a rule-based sensor fusion processing component.
33. The method of claim 25 , wherein the first sensor data and the second sensor data also correspond to the one or more first safety integrity levels.