Object detection and tracking using machine learning transformer models with attention
Object detection and tracking systems may use machine-learned transformer models with self-attention for detecting, classifying, and/or tracking objects in an environment. Techniques described herein may include receiving sensor data generated by different sensor modalities of a vehicle, determining different bounding shapes based on the different sensor modalities, and using a machine-learned transformer model to determine associated and/or combined bounding shapes. The machine-learned transformer model may receive a variable number of input bounding shapes representing any number of objects and various sensor modalities. Multiple stages of the transformer may be used to determine associated bounding shapes and to assign attributes for the associated bounding shapes, based on the individual bounding shapes of the different sensor modalities and/or previous bounding shapes for objects detected and tracked in a previous scene in the environment.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving first sensor data generated by a first modality of sensor of a vehicle, the first modality being lidar;
determining, based on the first sensor data, a first bounding shape representing an object in an environment at a first time;
receiving second sensor data generated by a second modality of sensor of the vehicle, the second modality being radar or vision;
determining, based on the second sensor data, a second bounding shape representing the object at the first time;
inputting the first bounding shape and the second bounding shape into a machine-learned transformer model, wherein the machine-learned transformer model is configured to:
receive a variable number of input bounding shapes, and
determine associations between the input bounding shapes;
determining, by the machine-learned transformer model and based at least in part on the first bounding shape and the second bounding shape, a combined bounding shape representing the object at the first time; and
controlling operation of the vehicle, based at least in part on the combined bounding shape representing the object.
2. The system of claim 1 , wherein the machine-learned transformer model is further configured to:
receive input representing an object track associated with a previous scene of the environment; and
determine the associations between the input bounding shapes, based at least in part on the object track.
3. The system of claim 1 , wherein the machine-learned transformer model includes a self-attention component configured to:
determine a first embedding associated with the first bounding shape; and
determine a second embedding associated with the second bounding shape.
4. The system of claim 1 , wherein the machine-learned transformer model is further configured to:
determining, in a first association stage, that the first bounding shape and the second bounding shape are associated with the object; and
determining, in a second classification stage after the first association stage, at least one of a classification, a yaw, or an intent associated with the object.
5. The system of claim 4 , wherein performing the first association stage in the machine-learned transformer model includes:
performing a first bounding shape association pass, based at least in part on a first set of bounding shapes;
determining, based on least in part on an output of the first bounding shape association pass, a residual subset of the first set of bounding shapes not associated during the first bounding shape association pass; and
performing a second bounding shape association pass after the first bounding shape association pass, based at least in part on the residual subset of the first set of bounding shapes.
6. A method comprising:
receiving first sensor data of an environment generated by a first modality of sensor;
determining, based on the first sensor data, a first geometric representation representing an object at a first time in the environment;
receiving second sensor data of the environment generated by a second modality of sensor;
determining, based on the second sensor data, a second geometric representation representing the object at the first time, wherein the second geometric representation is different from the first geometric representation;
inputting the first geometric representation and the second geometric representation into a machine-learned transformer model;
determining, by the machine-learned transformer model and based at least in part on the first geometric representation and the second geometric representation, a third geometric representation representing the object at the first time; and
controlling operation of a vehicle, based at least in part on the third geometric representation representing the object.
7. The method of claim 6 , wherein the machine-learned transformer model is configured to:
receive a non-fixed number of input geometric representations, and
output a non-fixed number of output geometric representations.
8. The method of claim 6 , wherein inputting the first geometric representation and the second geometric representation comprises:
providing, as input to the machine-learned transformer model, first input data representing the first geometric representation and second input data representing the geometric representation,
wherein the first input data comprises first geometric data associated with the first geometric representation and first classification data associated with the first geometric representation, and
wherein the second input data comprises second geometric data associated with the second geometric representation and second classification data associated with the second geometric representation.
9. The method of claim 6 , wherein the machine-learned transformer model is configured to:
determining, in a first association stage, that the first geometric representation and the second geometric representation are associated with the object; and
determining, in a second classification stage after the first association stage, at least one of a classification, a yaw, or an intent associated with the object.
10. The method of claim 9 , wherein performing the first association stage in the machine-learned transformer model includes:
performing a first geometric representation association pass, based at least in part on a first set of geometric representations;
determining, based on least in part on an output of the first geometric representation association pass, a residual subset of the first set of geometric representations not associated during the first geometric representation association pass; and
performing a second geometric representation association pass after the first geometric representation association pass, based at least in part on the residual subset of geometric representation.
11. The method of claim 6 , further comprising:
receiving third sensor data generated by a third modality of sensor;
determining, based on the third sensor data, a fourth geometric representation representing the object; and
inputting the fourth geometric representation, with the first geometric representation and the second geometric representation into the machine-learned transformer model,
wherein the first modality of sensor includes lidar, the second modality of sensor includes vision, and the third modality of sensor includes radar.
12. The method of claim 6 , wherein controlling the operation of the vehicle comprises:
rendering the third geometric representation representing the object into a top-down view of the environment; and
providing the top-down view of the environment as input to a neural network trained to output predicted state data associated with the object.
13. The method of claim 6 , wherein determining the third geometric representation further comprises inputting, to the machine-learned transformer model:
first data indicating a first sensor modality type associated with the first geometric representation; and
second data indicating a second sensor modality type associated with the second geometric representation.
14. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving first sensor data of an environment generated by a first modality of sensor;
determining, based on the first sensor data, a first geometric representation representing an object at a first time in the environment;
receiving second sensor data of the environment generated by a second modality of sensor;
determining, based on the second sensor data, a second geometric representation representing the object at the first time, wherein the second geometric representation is different from the first geometric representation;
inputting the first geometric representation and the second geometric representation into a machine-learned transformer model;
determining, by the machine-learned transformer model and based at least in part on the first geometric representation and the second geometric representation, a third combined geometric representation representing the object at the first time; and
controlling operation of a vehicle, based at least in part on the combined geometric representation representing the object.
15. The one or more non-transitory computer-readable media of claim 14 , wherein the machine-learned transformer model is configured to:
receive a non-fixed number of input geometric representations, and
output a non-fixed number of output geometric representations.
16. The one or more non-transitory computer-readable media of claim 14 , wherein inputting the first geometric representation and the second geometric representation comprises:
providing, as input to the machine-learned transformer model, first input data representing the first geometric representation and second input data representing the geometric representation,
wherein the first input data comprises first geometric data associated with the first geometric representation and first classification data associated with the first geometric representation, and
wherein the second input data comprises second geometric data associated with the second geometric representation and second classification data associated with the second geometric representation.
17. The one or more non-transitory computer-readable media of claim 14 , wherein the first geometric representation and the second geometric representation are associated with a scene of an environment at the first time, and
wherein the machine-learned transformer model is configured to receive additional input representing an object track associated with a previous scene of the environment at a previous time before the first time.
18. The one or more non-transitory computer-readable media of claim 14 , wherein the machine-learned transformer model includes a self-attention component configured to:
determine a first embedding associated with the first geometric representation; and
determine a second embedding associated with the second geometric representation.
19. The one or more non-transitory computer-readable media of claim 14 , wherein the machine-learned transformer model is configured to:
determining, in a first association stage, that the first geometric representation and the second geometric representation are associated with the object; and
determining, in a second classification stage after the first association stage, at least one of a classification, a yaw, or an intent associated with the object.
20. The one or more non-transitory computer-readable media of claim 19 , wherein performing the first association stage in the machine-learned transformer model includes:
performing a first geometric representation association pass, based at least in part on a first set of geometric representations;
determining, based on least in part on an output of the first geometric representation association pass, a residual subset of the first set of geometric representations not associated during the first geometric representation association pass; and
performing a second geometric representation association pass after the first geometric representation association pass, based at least in part on the residual subset of geometric representations.