Method for transforming 2D bounding boxes of objects into 3D positions for autonomous driving vehicles (ADVs)
In one embodiment, a system of an ADV perceives an object in a perspective view of the ADV using an image capturing device mounted on the ADV. The system generates a 2D bounding box for the object based on an image of the object captured by the image capturing device. The system determines an orientation and a 3D size of the object based on the image. The system projects onto the image a 3D representation of the object based on the orientation and the 3D size of the object. The system determines a distance from the ADV to the object based on the 3D representation of the object and the generated 2D bounding box. The system generates a trajectory in view of the determined distance from the ADV to the object to control the ADV to navigate the ADV around the object.
1. A computer-implemented method for operating an autonomous driving vehicle (ADV), the method comprising:
perceiving an object in a perspective view of the ADV using an image capturing device mounted on the ADV;
generating a 2D bounding box for the object based on an image of the object captured by the image capturing device;
classifying the object to be one of a plurality of known objects based on the image of the object;
determining an orientation and a size of the object based on the classification;
projecting onto the image a plurality of three-dimensional (3D) representations of the object having a similar orientation and size as the object, wherein the plurality of 3D representations represents the object with varying distances to the ADV;
determining a distance from the ADV to the object based on a dimension of a selected one from the plurality of 3D representations that approximately matches the dimension of the 2D bounding box; and
generating a trajectory in view of the determined distance from the ADV to the object to control the ADV to navigate the ADV around the object.
2. The computer-implemented method of claim 1 , wherein the 2D bounding box for the object is generated by:
applying a machine learning model to the image capturing the object; and
determining a length, a width, or an area of a 2D bounding box surrounding the object based on the machine learning model.
3. The computer-implemented method of claim 2 , further comprising:
searching a size for each of the plurality of 3D representations using a greedy search algorithm until a dimension of the projected 3D representation is within a predetermined threshold of the width or the height of the 2D bounding box; and
determining the distance from the ADV to the object based on the size.
4. The computer-implemented method of claim 2 , further comprising:
searching a size for each of the plurality of 3D representations using a binary search algorithm until a dimension of the projected 3D representation is within a predetermined threshold of the width, the height, or the area of the 2D bounding box; and
determining the distance from the ADV to the object based on the size.
5. The computer-implemented method of claim 1 , further comprises determining a center vector to the object based on the determined distance to the objected, wherein the center vector can be used to generate a driving trajectory to control the ADV.
6. The computer-implemented method of claim 5 , wherein determining a center vector of the object comprises:
shifting a 3D representation of the object left or right until a center of the 3D representation is approximately centered to a horizontal axis of a 2D center of the 2D bounding box;
shifting the 3D representation or down until the center of the 3D representation is approximately centered to a vertical axis of the 2D center of the 2D bounding box; and
determining the center vector based on the shifting of the 3D representation and the determined distance to the object.
7. The computer-implemented method of claim 6 , further comprises controlling the ADV based on the determined center vector to the object.
8. A non-transitory machine-readable medium having instructions stored therein, which when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:
perceiving an object in a perspective view of the ADV using an image capturing device mounted on the ADV;
generating a 2D bounding box for the object based on an image of the object captured by the image capturing device;
classifying the object to be one of a plurality of known objects based on the image of the object;
determining an orientation and a 3D size of the object based on the classification;
projecting onto the image a plurality of three-dimensional (3D) representations of the object having a similar orientation and size as the object, wherein the plurality of 3D representations represents the object with varying distances to the ADV;
determining a distance from the ADV to the object based on a dimension of a selected one from the plurality of 3D representations that approximately matches the dimension of the 2D bounding box; and
generating a trajectory in view of the determined distance from the ADV to the object to control the ADV to navigate the ADV around the object.
9. The non-transitory machine-readable medium of claim 8 , wherein the 2D bounding box for the object is generated by:
applying a machine learning model to the image capturing the object; and
determining a length, a width, or an area of a 2D bounding box surrounding the object based on the machine learning model.
10. The non-transitory machine-readable medium of claim 9 , further comprising:
searching a size for each of the plurality of 3D representations using a greedy search algorithm until a dimension of the 3D representation is within a predetermined threshold of the width or the height of the 2D bounding box; and
determining the distance from the ADV to the object based on the size.
11. The non-transitory machine-readable medium of claim 9 , further comprising:
searching a size for each of the plurality of 3D representations using a binary search algorithm until a dimension of the 3D representation is within a predetermined threshold of the width, the height, or the area of the 2D bounding box; and
determining the distance from the ADV to the object based on the size.
12. The non-transitory machine-readable medium of claim 8 , further comprises determining a center vector to the object based on the determined distance to the objected, wherein the center vector can be used to generate a driving trajectory to control the ADV.
13. The non-transitory machine-readable medium of claim 12 , wherein determining a center vector of the object comprises:
shifting a 3D representation of the object left or right until a center of the 3D representation is approximately centered to a horizontal axis of a 2D center of the 2D bounding box;
shifting the 3D representation up or down until the center of the 3D representation is approximately centered to a vertical axis of the 2D center of the 2D bounding box; and
determining the center vector based on the shifting of the 3D representation and the determined distance to the object.
14. The non-transitory machine-readable medium of claim 13 , further comprising controlling the ADV based on the determined center vector to the object.
15. A data processing system, comprising:
a plurality of processors; and
a memory coupled to the a plurality of processors to store instructions, which when executed by the a plurality of processors, cause the a plurality of processors to perform operations, the operations including
perceiving an object in a perspective view of the ADV using an image capturing device mounted on the ADV;
generating a 2D bounding box for the object based on an image of the object captured by the image capturing device;
classifying the object to be one of a plurality of known objects based on the image of the object;
determining an orientation and a 3D size of the object based on the classification;
projecting onto the image a plurality of three-dimensional (3D) representations of the object having a similar orientation and size as the object, wherein the plurality of 3D representations represents the object with varying distances to the ADV;
determining a distance from the ADV to the object based on a dimension of a selected one of the plurality of 3D representations that approximately matches the dimension of the 2D bounding box; and
generating a trajectory in view of the determined distance from the ADV to the object to control the ADV to navigate the ADV around the object.
16. The system of claim 15 , wherein the 2D bounding box for the object is generated by:
applying a machine learning model to the image capturing the object; and
determining a length, a width, or an area of a 2D bounding box surrounding the object based on the machine learning model.
17. The system of claim 16 , further comprising:
searching a size for each of the plurality of 3D representations using a greedy search algorithm until a dimension of the 3D representation is within a predetermined threshold of the width or the height of the 2D bounding box; and
determining the distance from the ADV to the object based on the size.
18. The system of claim 16 , further comprising:
searching a size for each of the plurality of 3D representations using a binary search algorithm until a dimension of the 3D representation is within a predetermined threshold of the width, the height, or the area of the 2D bounding box; and
determining the distance from the ADV to the object based on the size.
19. The system of claim 15 , further comprises determining a center vector to the object based on the determined distance to the objected, wherein the center vector can be used to generate a driving trajectory to control the ADV.
20. The system of claim 19 , wherein determining a center vector of the object comprises:
shifting a 3D representation of the object left or right until a center of the 3D representation is approximately centered to a horizontal axis of a 2D center of the 2D bounding box;
shifting the 3D representation up or down until the center of the 3D representation is approximately centered to a vertical axis of the 2D center of the 2D bounding box; and
determining the center vector based on the shifting of the 3D representation and the determined distance to the object.
21. The system of claim 20 , further comprising controlling the ADV based on the determined center vector to the object.