Sensor data fusion using cross-modal transformer
Techniques are generally described for fusing sensor data of different modalities using a transformer. In various examples, first sensor data may be received from a first sensor and second sensor data may be received from a second sensor. A first feature representation of the first sensor data may be generated using a first machine learning model and a second feature representation of the second sensor data may be generated using a second machine learning model. In some examples, a modified first feature representation of the first sensor data may be generated based at least in part on a self-attention mechanism of a transformer encoder. The modified first feature representation may be generated based at least in part on the first feature representation and the second feature representation. A computer vision task may be performed using the modified first feature representation.
1. A method comprising:
receiving first sensor data from a first sensor;
receiving second sensor data from a second sensor;
generating a first feature representation of the first sensor data using a first machine learning model;
generating a first position embedding for a first portion of the first feature representation, wherein the first position embedding relates the first portion of the first feature representation to a first spatial position of the first sensor data;
generating a second feature representation of the second sensor data using a second machine learning model;
generating a modified first feature representation of the first sensor data based at least in part on a self-attention mechanism of a transformer encoder, wherein the modified first feature representation represents a combination of the first feature representation with context from the second feature representation; and
performing a task using the modified first feature representation.
2. The method of claim 1 , wherein the first sensor comprises a first image sensor and the second sensor comprises a second image sensor, wherein the first image sensor and the second image sensor have different, overlapping fields of view with respect to one another.
3. The method of claim 1 , wherein:
the first sensor has a first modality;
the second sensor has a second modality different from the first modality; and
the modified first feature representation represents a combined feature representation of the first sensor data and the second sensor data.
4. The method of claim 1 , wherein the first sensor comprises a first image sensor and the second sensor comprises a second image sensor, wherein the first image sensor and the second image sensor have different, overlapping fields of view with respect to one another, the method further comprising:
detecting an object represented in the first sensor data based at least in part on a representation of the object in the second sensor data using the modified first feature representation.
5. The method of claim 1 , wherein the modified first feature representation represents, at least in part, a difference in a first field-of-view of the first sensor and a second field-of-view of the second sensor.
6. The method of claim 1 , wherein the task is a computer vision task related to autonomous navigation.
7. The method of claim 1 , further comprising:
generating the modified first feature representation of the first sensor data based at least in part on a plurality of sensors of different modalities, wherein the plurality of sensors comprises the first sensor, the second sensor, and at least one third sensor, and wherein the modified first feature representation represents attention calculated based at least in part on the first feature representation and the second feature representation.
8. The method of claim 1 , further comprising determining a semantic segmentation map corresponding to the first sensor data and the second sensor data.
9. A system comprising:
at least one processor;
a first sensor;
a second sensor; and
non-transitory computer-readable memory storing instructions that, when executed by the at least one processor, are effective to:
receive first sensor data from the first sensor;
receive second sensor data from the second sensor;
generate a first feature representation of the first sensor data using a first machine learning model;
generate a first position embedding for a first portion of the first feature representation, wherein the first position embedding relates the first portion of the first feature representation to a first spatial position of the first sensor data;
generate a second feature representation of the second sensor data using a second machine learning model;
generate a modified first feature representation of the first sensor data using a self-attention mechanism of a transformer encoder, wherein the modified first feature representation represents a combination of the first feature representation with context from the second feature representation; and
perform a task using the modified first feature representation.
10. The system of claim 9 , wherein the first sensor comprises a first image sensor and the second sensor comprises a second image sensor, wherein the first image sensor and the second image sensor have different, overlapping fields of view with respect to one another.
11. The system of claim 9 , wherein the first sensor comprises a first image sensor and the second sensor comprises a second image sensor, wherein the first image sensor and the second image sensor have different, overlapping fields of view with respect to one another, the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:
detect an object represented in the first sensor data based at least in part on a representation of the object in the second sensor data using the modified first feature representation.
12. The system of claim 9 , wherein:
the first sensor has a first modality;
the second sensor has a second modality different from the first modality; and
the modified first feature representation represents a combined feature representation of the first sensor data and the second sensor data.
13. The system of claim 9 , wherein the first sensor comprises a first image sensor and the second sensor comprises a second image sensor, wherein the first image sensor and the second image sensor have different, overlapping fields of view with respect to one another, wherein the non-transitory computer-readable memory stores further instructions that, when executed by the at least one processor, is further effective to:
detect an object represented in the first sensor data based at least in part on a representation of the object in the second sensor data using the modified first feature representation.
14. The system of claim 9 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to send the modified first feature representation to a transformer decoder, wherein the transformer decoder is configured to perform a computer vision task.
15. The system of claim 9 , the non-transitory computer-readable memory storing further instructions that, when executed by the at least one processor, are further effective to:
generate the modified first feature representation of the first sensor data based at least in part on a plurality of sensors of different modalities, wherein the plurality of sensors comprises the first sensor, the second sensor, and at least one third sensor, and wherein the modified first feature representation represents attention calculated based at least in part on the first feature representation and the second feature representation.
16. A method comprising:
receiving a first sensor data from a first image sensor, the first sensor data comprising a first frame of image data;
receiving second sensor data from a light detection and ranging (LIDAR) sensor, the second sensor data comprising three-dimensional data;
generating a first feature representation of the first frame of image data using a first machine learning model;
generating a first position embedding for a first portion of the first feature representation, wherein the first position embedding relates the first portion of the first feature representation to a first spatial position of the first sensor data;
generating a second feature representation of the three-dimensional data using a second machine learning model;
generating a modified first feature representation of the first sensor data based at least in part on a self-attention mechanism of a transformer encoder, wherein the modified first feature representation represents a combination of the first feature representation with context from the second feature representation; and
performing a task using the modified first feature representation.
17. The method of claim 16 , further comprising:
detecting an object represented in the first sensor data based at least in part on a representation of the object in the second sensor data using the modified first feature representation.
18. The method of claim 1 , wherein the first sensor data comprises three-dimensional (3D) point cloud data and wherein the first position embedding represents one or more points in the 3D point cloud data.
19. The system of claim 9 , wherein the first sensor data comprises three-dimensional (3D) point cloud data and wherein the first position embedding represents one or more points in the 3D point cloud data.
20. The method of claim 16 , wherein the first sensor data comprises three-dimensional (3D) point cloud data and wherein the first position embedding represents one or more points in the 3D point cloud data.