Joint 2D and 3D object tracking for autonomous systems and applications
In various examples, techniques for multi-dimensional tracking of objects using two-dimensional (2D) sensor data are described. Systems and methods may use first image data to determine a first 2D detected location and a first three-dimensional (3D) detected location of an object. The systems and methods may then determine a 2D estimated location using the first 2D detected location and a 3D estimated location using the first 3D detected location. The systems and methods may use second image data to determine a second 2D detected location and a second 3D detected location of a detected object, and may then determine that the object corresponds to the detected object using the 2D estimated location, the 3D estimated location, the second 2D detected location, and the second 3D detected location. The systems and method then generate, modify, delete, or otherwise update an object track that includes 2D state information and 3D state information.
1 . A method comprising:
detecting, based at least on sensor data, an object located within an environment;
determining, based at least on the sensor data, two-dimensional (2D) state information associated with the object;
determining, based at least on the sensor data, three-dimensional (3D) state information associated with the object;
performing tracking with respect to the object using the 2D state information associated with the object and the 3D state information associated with the object; and
generating, based at least on the tracking, a track associated with the object, the track being associated with the 2D state information associated with the object and the 3D state information associated with the object.
2 . The method of claim 1 , wherein the sensor data corresponds to one or more images, and wherein the method further comprises:
associating, based at least on the tracking, the object with one or more detected objects using a threshold number of the one or more images,
wherein the generating the track is based at least on the associating the object with the one or more detected objects using the threshold number of the one or more images.
3 . The method of claim 1 , further comprising:
determining, based at least on the tracking, a confidence score associated with the object; and
determining that the confidence score satisfies a threshold score,
wherein the generating the track is based at least on the confidence score satisfying the threshold score.
4 . The method of claim 3 , wherein the sensor data represents one or more images, and wherein the determining the confidence score associated with the object is based at least on one or more of:
one or more confidences associated with the one or more images depicting the object; or
one or more associations between the object and one or more objects detected using the one or more images.
5 . The method of claim 1 , further comprising:
determining, based at least on second sensor data, second 2D state information associated with a detected object;
determining, based at least on the second sensor data, second 3D state information associated with the detected object;
determining that the detected object corresponds to the object; and
updating, based at least on the detected object corresponding to the object, the track to be associated with the second 2D state information and the second 3D state information.
6 . The method of claim 1 , further comprising:
detecting, based at least on second sensor data, one or more objects;
determining that the object is not associated with the one or more objects; and
terminating, based at least on the object not being associated with the one or more objects, the track associated with the object.
7 . The method of claim 6 , further comprising:
determining, based at least on the object not being associated with the one or more objects, at least one of:
that a threshold number of images represented by the second sensor data do not depict the object; or
that a confidence score associated with the object does not satisfy a threshold confidence score,
wherein the terminating the track associated with the object is based at least on the at least one of the threshold number of images not depicting the object or the confidence score not satisfying the threshold confidence score.
8 . The method of claim 1 , wherein:
the 2D state information includes at least one of a bounding shape, a transition vector, or one or more feature descriptors; and
the 3D state information includes at least one of an object shape, a velocity, an acceleration, or one or more fence points.
9 . The method of claim 1 , further comprising causing, based at least on the track, a machine to navigate within the environment.
10 . A system comprising:
one or more processors to:
detect, based at least on first sensor data, an object located within an environment;
determine, based at least on the first sensor data, two-dimensional (2D) state information associated with the object;
determine, based at least on the first sensor data, three-dimensional (3D) state information associated with the object;
generate a track associated with the object, the track being associated with the 2D state information and the 3D state information;
detect, based at least on second sensor data, one or more objects;
determine that the object is not associated with the one or more objects; and
terminate, based at least on the object not being associated with the one or more objects, the track associated with the object.
11 . The system of claim 10 , wherein the first sensor data represents one or more images, and wherein the one or more processors are further to:
associate, based at least on the first sensor data, the object with one or more detected objects using a threshold number of the one or more images,
wherein the generation of the track is based at least on the object being associated with the one or more detected objects using the threshold number of the one or more images.
12 . The system of claim 10 , wherein the one or more processors are further to:
determine, based at least on the first sensor data, a confidence score associated with the object; and
determine that the confidence score satisfies a threshold score,
wherein the generation of the track is based at least on the confidence score satisfying the threshold score.
13 . The system of claim 12 , wherein the first sensor data represents one or more images, and wherein the determination of the confidence score associated with the object is based at least on one or more of:
one or more confidences associated with the one or more images depicting the object; or
one or more associations between the object and one or more objects detected using the one or more images.
14 . The system of claim 10 , wherein the one or more processors are further to:
determine, based at least on third sensor data obtained before the second sensor data, second 2D state information associated with a detected object;
determine, based at least on the third sensor data, second 3D state information associated with the detected object;
determine that the detected object corresponds to the object; and
update, based at least on the detected object corresponding to the object, the track to be associated with the second 2D state information and the second 3D state information.
15 . The system of claim 10 , wherein the one or more processors are further to:
determine, based at least on the object not being associated with the one or more objects, at least one of:
that a threshold number of images represented by the second sensor data do not depict the object; or
that a confidence score associated with the object does not satisfy a threshold confidence score,
wherein the termination of the track associated with the object is based at least on the at least one of the threshold number of images not depicting the object or the confidence score not satisfying the threshold confidence score.
16 . The system of claim 10 , wherein:
the 2D state information includes at least one of a bounding shape, a transition vector, or one or more feature descriptors; and
the 3D state information includes at least one of an object shape, a velocity, an acceleration, or one or more fence points.
17 . The system of claim 10 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system for generating or presenting at least one of mixed reality content, virtual reality content, or augmented reality content;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
18 . A processor comprising processing circuitry to:
determine, based at least on sensor data, two-dimensional (2D) state information associated with an object and three-dimensional (3D) state information associated with the object;
determine, using the 2D state information associated with the object and the 3D state information associated with the object, that the object corresponds to a tracked object; and
update, based at least on the object corresponding to the tracked object, a track associated with the tracked object to indicate the 2D state information and the 3D state information.
19 . The processor of claim 18 , wherein the processing circuitry is further to:
determine, based at least on second sensor data, second 2D state information and second 3D state information associated with one or more objects;
associate the tracked object with the one or more objects; and
update the track to be associated with the second 2D state information and the second 3D state information.
20 . The processor of claim 19 , wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system for generating or presenting at least one of mixed reality content, virtual reality content, or augmented reality content;
a system implemented using a robot;
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.