Tracking objects using sensor data segmentations and/or representations

1. A method comprising:

receiving, from one or more sensors, a first image comprising a representation of an object;

receiving, as a first segmentation, a first portion of the first image associated with the object, wherein the first segmentation is a subset of pixels of a region of interest associated with the object;

inputting, into a machine learned model, the first segmentation, the machine learned model being trained to output multi-dimensional vectors associated with an input segmentation;

receiving, from the machine learned model, a first multi-dimensional vector associated with the first segmentation;

receiving, from the one or more sensors, a second image;

receiving, as a second segmentation, a second portion of the second image associated with the object;

inputting, into the machine learned model, the second segmentation;

receiving, from the machine learned model, a second multi-dimensional vector associated with the second segmentation;

determining, based at least in part on the first multi-dimensional vector and the second multi-dimensional vector, a similarity between the first segmentation and the second segmentation;

based at least in part on the similarity, associating at least one of the first segmentation or the second segmentation with a track of the object; and

controlling an autonomous vehicle based at least in part on the track.

2. The method of claim 1 , wherein the first segmentation indicates fewer pixels than a region of interest associated with the object and the first image.

3. The method of claim 1 , wherein the machine learned model is trained, based at least in part on:

providing a first training segmentation to the machine learned model;

receiving, from the machine learned model, a first training multi-dimensional vector;

providing a second training segmentation to the machine learned model;

receiving, from the machine learned model, a second training multi-dimensional vector;

determining, based at least in part on the first training multi-dimensional vector and the second training multi-dimensional vector, a loss; and

altering one or more parameters associated with the machine learned model to minimize the loss.

4. The method of claim 1 , wherein:

the similarity comprises a distance between the first multi-dimensional vector and the second multi-dimensional vector; and

associating at least one of the first segmentation or the second segmentation with the track is based at least in part on determining that the distance is a shortest distance of a plurality of distances associated with one or more other segmentations.

5. The method of claim 1 , wherein:

the similarity comprises a distance between the first multi-dimensional vector and the second multi-dimensional vector; and

associating at least one of the first segmentation or the second segmentation with the track is based at least in part on determining that the distance is less than or equal to a distance threshold.

6. A system comprising:

one or more sensors;

one or more processors; and

a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:

receiving, from the one or more sensors, first sensor data;

inputting, into a machine learned model, a first portion of the first sensor data associated with an object, wherein the first portion includes up to an entirety of the first sensor data;

receiving, from the machine learned model, a first subset of data associated with the object and a first multi-dimensional vector associated with the first subset of data, wherein the first subset of data indicates a subset of a region of interest associated with the object;

receiving, from the one or more sensors, second sensor data;

inputting, into the machine learned model, a second portion of the second sensor data associated with the object, wherein the second portion includes up to an entirety of the second sensor data;

receiving, from the machine learned model, a second subset of data associated with the object and a second multi-dimensional vector associated with the second subset of data;

determining, based at least in part on the first multi-dimensional vector and the second multi-dimensional vector, a similarity between the first subset of data and the second subset of data; and

based at least in part on the similarity, associating at least one of the first subset of data or the second subset of data with a track of the object.

7. The system of claim 6 , wherein:

the first subset of data comprises a first segment and the second subset of data comprises a second subset of data, and

the machine learned model is trained based at least in part on:

providing a first training segmentation to the machine learned model;

receiving, from the machine learned model, a first training multi-dimensional vector;

providing a second training segmentation to the machine learned model;

receiving, from the machine learned model, a second training multi-dimensional vector;

determining, based at least in part on the first training multi-dimensional vector and the second training multi-dimensional vector, a loss; and

altering one or more parameters associated with the machine learned model to minimize the loss.

8. The system of claim 7 ,

wherein the machine learned model is a neural network,

wherein the first multi-dimensional vector includes a first multi-dimensional vector,

wherein n is an integer greater than two, and

wherein the second multi-dimensional vector includes a second multi-dimensional vector.

9. The system of claim 6 , wherein the machine learned model is a first machine learned model, and

wherein the operations further comprise:

inputting the first sensor data into a second machine learned model;

receiving, from the machine learned model, a first region of interest comprising the first portion of the first sensor data;

inputting the second sensor data into the second machine learned model; and

receiving, from the second machine learned model, a second region of interest comprising the first portion of sensor data, and

wherein:

the first subset of data corresponds to a subset of the first region of interest, and

the second subset of data corresponds to a subset of the second region of interest.

10. The system of claim 8 , wherein:

the similarity comprises a distance, in an embedding space, between the first multi-dimensional vector and the second multi-dimensional vector; and

associating at least one of the first portion or the second portion with the track is based at least in part on determining that the distance is less than or equal to a threshold distance.

11. The system of claim 6 , wherein at least one of the first sensor data or the second sensor data comprises one or more of:

image data,

lidar data,

RADAR data.

12. The system of claim 6 , wherein the operations further comprise determining the track based at least in part on the first portion and the second portion and wherein the track comprises at least one of:

a current position of at least part of a representation of the object in an image, the image being part of at least one of the first sensor data or the second sensor data;

one or more prior positions a representation of the object in the image;

a heading of the object;

a velocity of the object;

an acceleration of the object;

a jerk of the object; or

an uncertainty associated with the object.

13. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving first sensor data;

receiving a first subset of the first sensor data representing an object in an environment, the first subset indicating a portion of a region of interest associated with the object;

determining, by a machine-learned model, a first multi-dimensional vector associated with the first subset;

receiving second sensor data;

receiving a second subset of the second sensor data;

determining, by the machine-learned model, a second multi-dimensional vector associated with the second subset;

determining, based at least in part on the first multi-dimensional vector and the second multi-dimensional vector, a similarity between the first subset and the second subset; and

based at least in part on the similarity, associating at least one of the first subset or the second subset with a track of the object.

14. The non-transitory computer-readable medium of claim 13 ,

wherein determining, by a machine-learned model, the first multi-dimensional vector comprises:

inputting, into a neural network, the first subset; and

receiving, from the neural network, the first multi-dimensional vector, and

wherein determining the second multi-dimensional vector comprises:

inputting, into the neural network, the second subset; and

receiving, from the neural network, the second multi-dimensional vector.

15. The non-transitory computer-readable medium of claim 14 , wherein the neural network is trained, based at least in part, on:

providing a first training segmentation to the neural network;

receiving, from the neural network, a first training multi-dimensional vector;

providing a second training segmentation to the neural network;

receiving, from the neural network, a second training multi-dimensional vector;

determining, based at least in part on the first training multi-dimensional vector and the second training multi-dimensional vector, a loss; and

altering one or more parameters associated with the neural network to minimize the loss.

16. The non-transitory computer-readable medium of claim 14 , wherein:

the neural network is a first neural network, and

the operations further comprise:

inputting the first sensor data into a second neural network;

receiving, from the second neural network, the first subset;

inputting the second sensor data into the second neural network; and

receiving, from the second neural network, the second subset.

17. The non-transitory computer-readable medium of claim 13 , wherein:

the similarity comprises a Euclidian distance between the first multi-dimensional vector and the second multi-dimensional vector; and

associating at least one of the first subset or the second subset with the track is based at least in part on determining that the Euclidean distance is less than or equal to a threshold distance.

18. The non-transitory computer-readable medium of claim 16 , wherein

the first sensor data comprises first image data,

the first subset comprises a first image segment,

the second sensor data comprises second image data, and

the second subset comprises a second image segment.

19. The non-transitory computer-readable medium of claim 13 , wherein:

the first multi-dimensional vector is associated with a segmentation of the first sensor data;

the first data subset and the second data subset are different; and

the operations further comprising:

controlling, based at least in part on the track, a vehicle.

20. The non-transitory computer-readable medium of claim 13 , wherein the instructions further cause the one or more processors to determine, based at least in part on the first subset and the second subset, the track, wherein the track comprises at least one of:

a current position of at least part of a representation of the object in an image;

one or more prior positions a representation of the object in the image;

a heading of the object;

a velocity of the object;

an acceleration of the object

a jerk of the object; or

an uncertainty associated with the object.