Learning monocular 3D object detection from 2D semantic keypoint detection
A method for 3D object detection is described. The method includes detecting semantic keypoints from monocular images of a video stream capturing a 3D object. The method also includes inferring a 3D bounding box of the 3D object corresponding to the detected semantic vehicle keypoints. The method further includes scoring the inferred 3D bounding box of the 3D object. The method also includes detecting the 3D object according to a final 3D bounding box generated based on the scoring of the inferred 3D bounding box.
1. A method for 3D object detection, comprising:
detecting semantic keypoints from monocular images of a video stream capturing a 3D object;
inferring 3D bounding boxes of the 3D object by indexing the inferred 3D bounding box according to predicted keypoint coordinates corresponding to the detected semantic keypoints;
scoring the inferred 3D bounding boxes of the 3D object according to an objectness score, an object classification score, and 10D bounding box parameters predicted according to the predicted coordinates of the detected semantic keypoints;
discarding overlapping ones of the inferred 3D bounding boxes as redundant based on a user-defined overlap threshold using non-maxima suppression to determine a final set of 3D bounding boxes; and
detecting the 3D object according to the final set of 3D bounding boxes generated based on the scoring of the inferred 3D bounding boxes using score-thresholding and the non-maxima suppression.
2. The method of claim 1 , in which the semantic keypoints comprise universal vehicle keypoints.
3. The method of claim 2 , in which the universal vehicle keypoints comprise taillights, headlights, license plates, and wheel contact points to ground.
4. The method of claim 2 , further comprising labeling the universal vehicle keypoints in the images of the video stream during training.
5. The method of claim 2 , further comprising extracting, using a shared image backbone, the universal vehicle keypoints within the images of the video stream based on relevant appearance and geometric features of the images.
6. The method of claim 1 , further comprising planning a trajectory of an ego vehicle according to the detecting of the 3D object.
7. A non-transitory computer-readable medium having program code recorded thereon for 3D object detection, the program code being executed by a processor and comprising:
program code to detect semantic keypoints from monocular images of a video stream capturing a 3D object;
program code to infer 3D bounding boxes of the 3D object by indexing the inferred 3D bounding box according to predicted keypoint coordinates corresponding to the detected semantic keypoints;
program code to score the inferred 3D bounding boxes of the 3D object according to an objectness score, an object classification score, and 10D bounding box parameters predicted according to the predicted coordinates of the detected semantic keypoints;
program code to discard overlapping ones of the inferred 3D bounding boxes as redundant based on a user-defined overlap threshold using non-maxima suppression to determine a final set of 3D bounding boxes; and
program code to detect the final set of 3D bounding boxes generated based on the scoring of the inferred 3D bounding boxes using score-thresholding and the non- maxima suppression.
8. The non-transitory computer-readable medium of claim 7 , in which the semantic keypoints comprise universal vehicle keypoints.
9. The non-transitory computer-readable medium of claim 8 , in which the universal vehicle keypoints comprise taillights, headlights, license plates, and wheel contact points to ground.
10. The non-transitory computer-readable medium of claim 8 , further comprising program code to label the universal vehicle keypoints in the images of the video stream during training.
11. The non-transitory computer-readable medium of claim 8 , further comprising program code to extract, using a shared image backbone, the universal vehicle keypoints within the images of the video stream based on relevant appearance and geometric features of the images.
12. The non-transitory computer-readable medium of claim 7 , further comprising program code to plan a trajectory of an ego vehicle according to the detecting of the 3D object.
13. A system for 3D object detection, the system comprising:
a semantic keypoint detection module to detect semantic keypoints from monocular images of a video stream capturing a 3D object;
a 3 D bounding box module to infer 3D bounding boxes of the 3D object by indexing the inferred 3D bounding box according to predicted keypoint coordinates corresponding to the detected semantic keypoints;
a bounding box scoring module to score the inferred 3D bounding boxes of the 3D object according to an objectness score, an object classification score, and 10D bounding box parameters predicted according to the predicted coordinates of the detected semantic keypoints; and
a 3D object detector model to discard overlapping ones of the inferred 3D bounding boxes as redundant based on a user-defined overlap threshold using non-maxima suppression to determine a final set of 3D bounding boxes, and to detect the 3D object according to the non-maxima suppression generated based on the scoring of the inferred 3D bounding boxes using score-thresholding and the non-maxima suppression.
14. The system of claim 13 , further comprising a planner module to plan a trajectory of an ego vehicle according to detection of the 3D object.