Tracking objects using sensor data segmentations and/or representations
Techniques are disclosed for tracking objects in sensor data, such as multiple images or multiple LIDAR clouds. The techniques may include comparing segmentations of sensor data such as by, for example, determining a similarity of a first segmentation of first sensor data and a second segmentation of second sensor data. Comparing the similarity may comprise determining a first embedding associated with the first segmentation and a second embedding associated with the second segmentation and determining a distance between the first embedding and the second embedding. The techniques may improve the accuracy and/or safety of systems integrating the techniques discussed herein.
1. A method comprising:
receiving first sensor data comprising a representation of an object, wherein the first sensor data is associated with a first sensor type;
receiving a first sensor data segmentation of the first sensor data associated with the object;
inputting, into a machine learned model, the first sensor data segmentation and a semantic label associated with the first sensor data segmentation, the machine learned model being trained to output multi-dimensional vectors associated with image data;
receiving, from the machine learned model, a first multi-dimensional vector associated with the first sensor data segmentation and the semantic label;
receiving second sensor data, wherein the second sensor data is associated with a second sensor type;
receiving a second sensor data segmentation of the second sensor data associated with the object;
inputting, into the machine learned model, the second sensor data segmentation;
receiving, from the machine learned model, a second multi-dimensional vector associated with the second sensor data segmentation;
determining, based at least in part on the first multi-dimensional vector and the second multi-dimensional vector, a degree of similarity between the first sensor data segmentation and the second sensor data segmentation; and
controlling an autonomous vehicle based at least in part on the degree of similarity.
2. The method of claim 1 , wherein the machine learned model is a first machine learned model and wherein the method further comprises:
determining the first sensor data segmentation by a second machine learned model based at least in part on the first sensor data; and
determining the second sensor data segmentation by the second machine learned model based at least in part on the second sensor data.
3. The method of claim 1 , further comprising associating, based at least in part on the similarity, at least one of the first sensor data segmentation or the second sensor data segmentation with the object.
4. The method of claim 1 , wherein determining the degree of similarity comprises at least one of comparing or determining a distance between the first multi-dimensional vector and the second multi-dimensional vector.
5. The method of claim 1 , wherein the first multi-dimensional vector is an embedding.
6. A device comprising:
one or more processors; and
a memory storing processor-executable instructions that, when executed by the one or more processors, cause the device to perform operations comprising:
receiving first sensor data associated with a first sensor modality;
inputting, into a machine learned model, a first portion of the first sensor data;
receiving, from the machine learned model, a first multi-dimensional vector associated with the first portion;
receiving second sensor data associated with a second sensor modality, wherein:
the first sensor data is associated with a first sensor type,
the second sensor data is associated with a second sensor type, and
the second sensor modality is a different modality than the first sensor modality;
inputting, into the machine learned model, a second portion of the second sensor data;
receiving, from the machine learned model, a second multi-dimensional vector associated with the second portion;
determining, based at least in part on the first multi-dimensional vector and the second multi-dimensional vector, a degree of similarity between the first portion and the second portion; and
controlling an autonomous vehicle based at least in part on the degree of similarity.
7. The device of claim 6 , wherein the operations further comprise receiving, from the machine learned model, a first subset of data associated with an object, wherein the first subset of data includes a region of interest associated with the object in the first sensor data.
8. The device of claim 6 , wherein the operations further comprise:
determining, by a second machine learned model and based at least in part on the first sensor data, a confidence that the first portion of the first sensor data is associated with an object; and
determining, by the second machine learned model and based at least in part on the second sensor data, a confidence that the second portion of the second sensor data is associated with the object.
9. The device of claim 6 , wherein the machine learned model is a first machine learned model and wherein the operations further comprise:
determining the first portion by a second machine learned model based at least in part on the first sensor data, wherein the first portion is a first segmentation of the first sensor data; and
determining the second portion by the second machine learned model based at least in part on the second sensor data, wherein the second portion is a second segmentation of the second sensor data.
10. The device of claim 9 , wherein determining the degree of similarity comprises at least one of comparing or determining a distance between the first multi-dimensional vector and the second multi-dimensional vector.
11. The device of claim 6 , wherein the operations further comprise associating, based at least in part on the degree of similarity, the first portion with the second portion and an object represented in at least one of the first sensor data or the second sensor data.
12. The device of claim 6 , wherein:
the operations further comprise associating, based at least in part on the degree of similarity, the first portion and the second portion with a track; and
the track comprises at least one of: a current position of at least part of a representation of an object in the first sensor data or the second sensor data; one or more prior positions a representation of the object in previous sensor data; a heading of the object; a velocity of the object; an acceleration of the object; a jerk of the object; or an uncertainty associated with the object.
13. The device of claim 12 , wherein controlling the autonomous vehicle is further based at least in part on the track.
14. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving first sensor data associated with a first sensor type;
inputting, into a machine learned model, a first sensor data segmentation of the first sensor data and a semantic label associated with the first sensor data segmentation, the first sensor data segmentation comprising a sensor data segmentation of the first sensor data;
receiving, from the machine learned model, a first multi-dimensional vector associated with the first sensor data segmentation and the semantic label;
receiving second sensor data associated with a second sensor type;
inputting, into the machine learned model, a second sensor data segmentation of the second sensor data;
receiving, from the machine learned model, a second multi-dimensional vector associated with the second sensor data segmentation; determining, based at least in part on the first multi-dimensional vector and the second multi-dimensional vector, a degree of similarity between the first sensor data segmentation and the second sensor data segmentation; and
controlling an autonomous vehicle based at least in part on the degree of similarity.
15. The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise receiving, from the machine learned model, a first subset of data associated with an object, wherein the first subset of data includes a region of interest associated with the object in the first sensor data.
16. The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise:
determining, by a second machine learned model and based at least in part on the first sensor data, a confidence that the first sensor data segmentation of the first sensor data is associated with an object; and
determining, by the second machine learned model and based at least in part on the second sensor data, a confidence that the second sensor data segmentation of the second sensor data is associated with the object.
17. The non-transitory computer-readable medium of claim 14 , wherein the machine learned model is a first machine learned model and wherein the operations further comprise:
determining the first sensor data segmentation by a second machine learned model based at least in part on the first sensor data; and
determining the second sensor data segmentation by the second machine learned model based at least in part on the second sensor data.
18. The non-transitory computer-readable medium of claim 17 , wherein determining the degree of similarity comprises at least one of comparing or determining a distance between the first multi-dimensional vector and the second multi-dimensional vector.
19. The non-transitory computer-readable medium of claim 14 , wherein the operations further comprise associating, based at least in part on the degree of similarity, the first sensor data segmentation with the second sensor data segmentation and an object represented in at least one of the first sensor data or the second sensor data.
20. The non-transitory computer-readable medium of claim 14 , wherein:
the operations further comprise associating, based at least in part on the degree of similarity, the first sensor data segmentation and the second sensor data segmentation with a track; and
the track comprises at least one of: a current position of at least part of a representation of an object in the first sensor data or the second sensor data; one or more prior positions a representation of the object in previous sensor data; a heading of the object; a velocity of the object; an acceleration of the object; a jerk of the object; or an uncertainty associated with the object.