Systems and methods for controlling a vehicle by detecting and tracking objects through associated detections
This disclosure provides methods and systems for dynamically detecting and tracking objects in an environment of an autonomous vehicle. In some embodiments, the method comprises: receiving image data from sensors of the autonomous vehicle, the image data comprising a plurality of images representative of objects in a field of view of the autonomous vehicle; detecting the objects in the plurality of images through an object detector; generating image embeddings for the objects detected in the plurality of images; determining similarity scores of the image embeddings of the objects that are detected in images received from two or more different sensors; identifying the objects that are detected in the images received from the two or more different sensors as a candidate object for tracking, if the objects have a similarity score of the image embeddings equal to or greater than a threshold value; and initializing a track for the candidate object.
1 . A method of dynamically detecting and tracking objects in an environment of an autonomous vehicle, comprising:
receiving image data from sensors of the autonomous vehicle, the image data comprising a plurality of images representative of objects in a field of view of the autonomous vehicle; and
using a processor:
detecting the objects in the plurality of images through an object detector;
generating image embeddings for the objects detected in the plurality of images;
determining similarity scores of the image embeddings of the objects that are detected in images received from two or more different sensors;
identifying the objects that are detected in the images received from the two or more different sensors as a candidate object for tracking, if the objects have a similarity score of the image embeddings equal to or greater than a threshold value; and
initializing a track for the candidate object.
2 . The method of claim 1 , wherein the step of generating the image embeddings comprises generating the image embeddings by an embedding encoder trained using contrastive representation learning.
3 . The method of claim 1 , further comprising: generating additional image embeddings for objects detected in a second set of images received at a different time; and determining similarity scores between the image embeddings and the additional image embeddings.
4 . The method of claim 3 , comprising performing incremental clustering to determine the similarity scores between existing image embeddings and new embeddings.
5 . The method of claim 3 , comprising: determining a second similarity score between an image embedding of the candidate object and the additional image embeddings; and updating the image embedding of the candidate object if the second similarity score is equal to or greater than the threshold value.
6 . The method of claim 1 , wherein the sensors comprise two or more cameras.
7 . The method of claim 1 , comprising preprocessing the images prior to detecting the objects.
8 . The method of claim 1 , wherein the image embeddings are represented by feature vectors.
9 . The method of claim 1 , wherein the object detector comprises a neural network object detector.
10 . The method of claim 1 , wherein the step of determining the similarity scores of image the embeddings of the objects comprising determining the similarity scores of image the embeddings of the objects by clustering using the DBSCAN algorithm.
11 . A system for dynamically detecting and tracking objects in an environment of an autonomous vehicle, comprising:
a plurality of sensors, configured to receive image data comprising a plurality of images representative of objects in a field of view of the autonomous vehicle; and
a processor, configured to:
detect the objects in the plurality of images through an object detector;
generate image embeddings for the objects detected in the plurality of images;
determine similarity scores of the image embeddings of the objects that are detected in images received from two or more different sensors;
identify the objects that are detected in the images received from the two or more different sensors as a candidate object for tracking, if the objects have a similarity score of the image embeddings equal to or greater than a threshold value; and
initialize a track for the candidate object.
12 . The system of claim 11 , wherein the processor is configured to generate the image embeddings by an embedding encoder trained using contrastive representation learning.
13 . The system of claim 11 , wherein the processor is further configured to generate additional image embeddings for objects detected in a second set of images received at a different time, and determine similarity scores between the image embeddings and the additional image embeddings.
14 . The system of claim 13 , wherein the processor is configured to perform incremental clustering to determine the similarity scores between existing image embeddings and new embeddings.
15 . The system of claim 13 , wherein the processor is configured to determine a second similarity score between an image embedding of the candidate object and the additional image embeddings, and update the image embedding of the candidate object if the second similarity score is equal to or greater than the threshold value.
16 . The system of claim 11 , wherein the sensors comprise two or more cameras.
17 . The system of claim 11 , wherein the processor is configured to preprocess the images prior to detecting the objects.
18 . The system of claim 11 , wherein the image embeddings are represented by feature vectors.
19 . The system of claim 11 , wherein the object detector comprises a neural network object detector.
20 . The system of claim 11 , wherein the processor is configured to determine the similarity scores of image the embeddings of the objects by clustering using the DBSCAN algorithm.