SENSOR FUSION FOR AUTONOMOUS MACHINE APPLICATIONS USING MACHINE LEARNING
In various examples, a multi-sensor fusion machine learning model—such as a deep neural network (DNN)—may be deployed to fuse data from a plurality of individual machine learning models. As such, the multi-sensor fusion network may use outputs from a plurality of machine learning models as input to generate a fused output that represents data from fields of view or sensory fields of each of the sensors supplying the machine learning models, while accounting for learned associations between boundary or overlap regions of the various fields of view of the source sensors. In this way, the fused output may be less likely to include duplicate, inaccurate, or noisy data with respect to objects or features in the environment, as the fusion network may be trained to account for multiple instances of a same object appearing in different input representations.
1 . A method comprising:
generating a first output using one or more first layers of one or more neural networks and based at least on first sensor data obtained using one or more first sensors;
generating a second output using one or more second layers of the one or more neural networks and based at least on second sensor data obtained using one or more second sensors;
generating a fused output using one or more fusion layers of the one or more neural networks and based at least on the first output and the second output; and
performing one or more operations associated with a machine based at least on the fused output.
2 . The method of claim 1 , wherein the one or more first sensors include a same type of sensor as the one or more second sensors.
3 . The method of claim 1 , wherein:
the one or more first sensors include a first type of sensor; and
the one or more second sensors include a second type of sensor that is different from the first type of sensor.
4 . The method of claim 1 , wherein:
the first output corresponds to a first field of view or a first sensory field associated with the one or more first sensors;
the second output corresponds to a second field of view or a second sensory field associated with the one or more second sensors; and
the fused output represents a fused field of view or a fused sensory field that includes a portion of the first field of view or a first portion of the sensory field and a first portion of the second field of view or a first portion of the second sensory field.
5 . The method of claim 1 , wherein:
the first output is associated with a first representation of an object;
the second output is associated with a second representation of the object; and
the fused output is associated with a fused representation of the object that is based at least on the first representation and the second representation.
6 . The method of claim 1 , wherein:
the first output is associated with first information corresponding to an object;
the second output is associated with second information corresponding to the object; and
the fused output is associated with fused information corresponding to the object that is based at least on the first information and the second information.
7 . The method of claim 1 , further comprising:
receiving an input representative of at least one of: one or more probability distribution representations, one or more velocity representations, one or more object instance representations, or one or more object appearance representations,
wherein the generating the fused output is further based at least on the input.
8 . The method of claim 1 , wherein:
the one or more first sensors include one or more first fields of view or one or more first sensory fields; and
the one or more second sensors include one or more second fields of view or one or more second sensory fields that at least partially overlap with the one or more first fields of view or the one or more first sensory fields.
9 . A system comprising:
one or more processors to:
obtain first data generated using one or more first layers of a machine learning model and based at least on first sensor data obtained using one or more first sensors;
obtain second data generated using one or more second layers of the machine learning model and based at least on second sensor data obtained using one or more second sensors;
generate fused data using one or more fusion layers of the machine learning model and based at least on the first data and the second data; and
cause performance of one or more operations associated with a machine based at least on the fused data.
10 . The system of claim 9 , wherein the one or more first sensors include a same type of sensor as the one or more second sensors.
11 . The system of claim 9 , wherein:
the one or more first sensors include a first type of sensor; and
the one or more second sensors include a second type of sensor that is different from the first type of sensor.
12 . The system of claim 9 , wherein:
the first data corresponds to a first field of view or a first sensory field associated with the one or more first sensors;
the second data corresponds to a second field of view or a second sensory field associated with the one or more second sensors; and
the fused data represents a combined field of view or a combined sensory field that includes a portion of the first field of view or a portion of the first sensory field and a portion of the second field of view or a portion of the second sensory field.
13 . The system of claim 9 , wherein:
the first data corresponds to a first representation of an object;
the second data corresponds to a second representation of the object; and
the fused data corresponds to a fused representation of the object that is based at least on the first representation and the second representation.
14 . The system of claim 9 , wherein:
the first data represents first information corresponding to an object;
the second data represents second information corresponding to the object; and
the fused data represents fused information corresponding to the object that is based at least on the first information and the second information.
15 . The system of claim 9 , wherein the one or more processors are further to:
obtain input data representative of at least one of: one or more probability distribution representations, one or more velocity representations, one or more object instance representations, or one or more object appearance representations,
wherein the fused data is further generated based at least on the input data.
16 . The system of claim 9 , wherein the one or more fusion layers are subsequent the one or more first layers and the one or more second layers in an architecture of the machine learning model.
17 . The system of claim 9 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
18 . One or more processors comprising:
processing circuitry to cause a machine to perform one or more operations based at least on a fused output generated using one or more fusion layers of a neural network, wherein the one or more fusion layers generate the fused output based at least on processing a first output generated using one or more first layers of the neural network and a second output generating using one or more second layers of the neural, wherein the one or more first layers process sensor data of a different sensor modality than the one or more second layers.
19 . The one or more processors of claim 18 , wherein the fused output represents a top-down representation of an environment that includes at least first information associated with the first output and second information associated with the second output.
20 . The one or more processors of claim 19 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.