Feature detection and localization
In various examples, three-dimensional (3D) object or feature detection and localization for autonomous and semi-autonomous systems and applications is described herein. Systems and methods are disclosed that use different types of sensors, such as an image sensor and a LIDAR sensor, to determine information associated with objects, such as traffic objects (e.g., traffic signs, traffic signals, traffic markings, etc.). To determine the information for an object, image data is processed to determine a bounding shape associated with the object. The bounding shape is then used to determine a 3D shape, such as a frustum, corresponding to the object. Additionally, points data generated using the LIDAR sensor, such as an occupancy map and/or a point cloud, is processed to identify a portion of the points associated with (e.g., located within) the 3D shape. This portion of the points may then be used to determine the information associated with the object.
1 . A method comprising:
determining, based at least on image data representative of an image, a two-dimensional (2D) bounding shape associated with a traffic signal depicted in the image;
determining, based at least on the 2D bounding shape, a three-dimensional (3D) bounding shape associated with the traffic signal;
determining, based at least on one or more points from a point cloud being located within the 3D bounding shape, a third plane that is associated with the traffic signal;
determining a first geometry that includes at least a portion of the third plane;
determining a second geometry associated with the traffic signal based on at least one of a number of one or more indicators associated with the traffic signal or an orientation of the one or more indicators; and
determining a 3D location associated with the traffic signal by at least adjusting the first geometry based at least on the second geometry associated with the traffic signal.
2 . The method of claim 1 , wherein the determining the third plane associated with the traffic signal comprises:
determining, based at least on one or more locations associated with the one or more points, a center of the one or more points; and
determining that the third plane is located at the center of the one or more points, the center of the one or more points being located within the 3D bounding shape.
3 . The method of claim 1 , wherein the determining the first geometry is based at least on projecting one or more first vertices associated with the 2D bounding shape to one or more second vertices associated with the third plane.
4 . The method of claim 1 , further comprising:
determining a direction of travel associated with a machine; and
determining, based at least on the direction of travel, an orientation associated with the traffic signal.
5 . The method of claim 4 , wherein the determining the orientation associated with the traffic signal comprises:
determining that the direction of travel includes a first direction; and
determining, based at least on the first direction, the orientation associated with the traffic signal as including a second direction that is opposite to the first direction.
6 . The method of claim 1 , further comprising:
determining the at least one of the number of the one or more indicators associated with the traffic signal or the orientation of the one or more indicators associated with the traffic signal.
7 . The method of claim 1 , wherein the determining the 3D bounding shape associated with the traffic signal comprises determining, based at least on the 2D bounding shape, a frustum associated with the traffic signal, the frustum including the 3D bounding shape.
8 . The method of claim 1 , wherein the determining the 3D bounding shape associated with the traffic signal comprises:
determining a second plane based at least on the 2D bounding shape and a first projection distance;
determining a third plane based at least on the 2D bounding shape and a second projection distance; and
determining the 3D bounding shape based at least on the second plane and the third plane.
9 . A system comprising:
one or more processors configured to:
determine a two-dimensional (2D) bounding shape associated with an object depicted in an image;
determine, based at least on the 2D bounding shape, a three-dimensional (3D) bounding shape associated with the object within an environment;
determine, based at least on one or more points from a point cloud that are located within the 3D bounding shape, a first geometry that includes at least a portion of a plane that is located within the 3D bounding shape;
determine a second geometry associated with the object based on at least a number of one or more indicators associated with the object or an orientation of the one or more indicators; and
determine a 3D location associated with the object by at least adjusting the first geometry based at least on the second geometry associated with the object.
10 . The system of claim 9 , wherein the determination of the first geometry comprises:
determining, based at least on one or more locations associated with the one or more points, a center of the one or more points;
determining, based at least on the center of the one or more points, the plane associated with the object that is located within the 3D bounding shape; and
determining the first geometry as including the at least the portion of the plane.
11 . The system of claim 9 , wherein the one or more processors are further configured to:
project one or more first vertices associated with the 2D bounding shape to one or more second vertices associated with the plane,
wherein the first geometry is further determined based at least on the one or more second vertices.
12 . The system of claim 9 , wherein the one or more processors are further configured to:
determine a direction of travel associated with a machine; and
determine, based at least on the direction of travel, an orientation associated with the object.
13 . The system of claim 9 , wherein the one or more processors are further configured to:
determine the at least one of the number of the one or more indicators associated with the object or the orientation of the one or more indicators associated with the object.
14 . The system of claim 9 , wherein the determination of the 3D bounding shape associated with the object comprises determining, based at least on the 2D bounding shape, a frustum associated with the object, the frustum including the 3D bounding shape.
15 . The system of claim 9 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system implementing one or more large language models (LLMs);
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
16 . One or more processors comprising processing circuitry to:
determine, based at least on image data representative of an image, a bounding shape associated with a traffic signal depicted in the image;
determine, based at least on the bounding shape, one or more points from a point cloud that correspond to the traffic signal;
determine, based at least on the one or more points, a plane associated with the traffic signal;
determine a first geometry associated with at least a portion of the plane;
determine a second geometry associated with the traffic signal based on at least one of an orientation of one or more indicators of the traffic signal or a number of the one or more indicators; and
determine a three-dimensional (3D) location associated with the traffic signal by adjusting the first geometry associated with the at least the portion of the plane based at least on the second geometry associated with the traffic signal.
17 . The one or more processors of claim 16 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system implementing one or more large language models (LLMs);
a system for performing conversational AI operations;
a system for generating synthetic data;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
18 . The one or more processors of claim 16 , wherein the processing circuitry is further to:
determine a 3D bounding shape based at least on the bounding shape,
wherein the determining the one or more points that correspond to the traffic signal is based at least on the 3D bounding shape.
19 . The one or more processors of claim 18 , wherein the plane is determined as being located within the 3D bounding shape.
20 . The one or more processors of claim 18 , wherein the first geometry associated with the at least the portion of the plane is determined based at least on projecting one or more first vertices associated with the bounding shape to determine one or more second vertices associated with the plane.