Monocular 3D object detection from image semantics network
Techniques are provided for monocular 3D object detection from an image semantics network. An image semantics network (ISN) is a single stage, single image object detection network that is based on single shot detection (SSD). In an embodiment, the ISN augments the SSD outputs to provide encoded 3D properties of the object along with a 2D bounding box and classification scores. For each priorbox, a 3D bounding box is generated for the object using the dimensions and location of the priorbox, the encoded 3D properties and camera intrinsic parameters.
1. A method comprising:
receiving, using one or more processors of a vehicle, images from a camera of the vehicle;
generating, using an object detection network with the images as input, two-dimensional (2D) positions, dimensions and center offsets of 2priorboxes and corresponding classification scores for each object detected in the images, wherein the object detection network is a single stage, single image network with a single shot detector detection head;
for each detected object:
generating encoded three-dimensional (3D) properties of the detected object, the encoded 3D properties including a center projection of a 3D bounding box for the detected object, the center projection determined by the position, dimensions and center offsets of a 2D bounding box corresponding to the detected object;
generating a 3D bounding box for the detected object using the encoded 3D properties and camera intrinsic parameters;
computing a route or trajectory for the vehicle using at least in part the generated 3D bounding boxes; and
causing, using a controller of the vehicle, the vehicle to travel along the route or trajectory.
2. The method of claim 1 , wherein the encoded 3D properties include dimensions, radial distance, viewing angle and center projection offsets.
3. The method of claim 1 , wherein generating, using the one or more processors, encoded three-dimensional (3D) properties of the object, further comprises: for each priorbox, estimating a vector of parameters that include a set of offsets from a bottom center of the priorbox, width, length and height of the object, radial distance from a center of the camera to the object center and viewing angle.
4. The method of claim 1 , wherein the object detection network outputs six groups of parameters including classification, 2D localization, dimensional, orientation, radial distance and center projection, and the method further comprises:
computing a loss function for each group of parameters individually; and
using the loss functions to train the object detection network.
5. A system comprising:
one or more processors;
memory storing instructions that when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving images from a camera of a vehicle;
generating, using an object detection network with the images as input, two-dimensional (2D) positions, dimensions and center offsets of the 2D priorboxes and corresponding classification scores for each object detected in the images, wherein the object detection network is a single stage, single image network with a single shot detector detection head;
for each detected object:
generating encoded three-dimensional (3D) properties of the detected object, the encoded 3D properties including a center projection of a 3D bounding box for the detected object, the center projection determined by the position, dimensions and center offsets of a corresponding 2D bounding box;
generating a 3D bounding box for the detected object using the encoded 3D properties and camera intrinsic parameters;
computing a route or trajectory for the vehicle using at least in part the generated 3D bounding boxes; and
causing, using a controller of the vehicle, the vehicle to travel along the route or trajectory.
6. The system of claim 5 , wherein the encoded 3D properties include dimensions, radial distance, viewing angle and center projection offsets for a 3D bounding box.
7. The system of claim 5 , wherein generating encoded three-dimensional (3D) properties of the object, further comprises: for each priorbox, estimating a vector of parameters that include a set of offsets from a bottom center of the priorbox, width, length and height of the object, radial distance from a center of the camera to the object center and viewing angle.
8. The system of claim 5 , wherein the object detection network outputs six groups of parameters including classification, 2D localization, dimensional, orientation, radial distance and center projection of a 3D bounding box, and the method further comprise:
computing a loss function for each group of parameters individually; and
using the loss functions to train the object detection network.
9. A non-transitory, computer-readable storage medium having instructions stored thereon, that when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving images from a camera of a vehicle;
generating, using an object detection network with the images as input, two-dimensional (2D) positions, dimensions and center offsets of 2D priorboxes and corresponding classification scores for each object detected in the images, wherein the object detection network is a single stage, single image network with a single shot detector detection head;
for each object:
generating encoded three-dimensional (3D) properties of the object, the encoded 3D properties including a center projection of a 3D bounding box for the detected object, the center projection determined by the position, dimensions and center offsets of a corresponding 2D bounding box;
generating a 3D bounding box for the detected object using the encoded 3D properties and camera intrinsic parameters;
computing a route or trajectory for the vehicle using at least in part the generated 3D bounding boxes; and
causing, using a controller of the vehicle, the vehicle to travel along the route or trajectory.
10. The non-transitory, computer-readable storage medium of claim 9 , wherein the encoded 3D properties include dimensions, radial distance, viewing angle and center projection offsets of a 3D bounding box.
11. The non-transitory, computer-readable storage medium of claim 9 , wherein generating, using the one or more processors, encoded three-dimensional (3D) properties of the object, further comprises: for each priorbox, estimating a vector of parameters that include a set of offsets from a bottom center of the priorbox, width, length and height of the object, radial distance from a center of the camera to the object center and viewing angle.
12. The non-transitory, computer-readable storage medium of claim 9 , wherein the object detection network outputs six groups of parameters including classification, 2D localization, dimensional, orientation, radial distance and center projection, and the operations further comprises:
computing a loss function for each group of parameters individually; and
using the loss functions to train the object detection network.