Joint 3D detection and segmentation using bird's eye view and perspective view
An image processing method includes performing, using images obtained from one or more sensors onboard a vehicle, a 2-dimensional (2D) feature extraction; performing, a 3-dimensional (3D) feature extraction on the images; detecting objects in the images by fusing detection results from the 2D feature extraction and the 3D feature extraction.
1 . A method of detecting objects in sensor data, comprising:
performing, using images obtained from one or more sensors onboard a vehicle, a 2-dimensional (2D) feature extraction;
performing, a 3-dimensional (3D) feature extraction on the images;
detecting objects in the images by fusing detection results from the 2D feature extraction and the 3D feature extraction;
refining 3D feature estimates using dual-space object queries that include joint proposals formed based on 2D features resulting from the 2D feature extraction and 3D features resulting from the 3D feature extraction;
wherein the refining comprises performing a multi-level refinement wherein, at each layer of the multi-level refinement, a self-attention layer that acts on both the 2D features and the 3D features, a first cross-attention layer that acts only on the 2D features, and a second cross-attention layer that acts only on the 3D features are used.
2 . The method of claim 1 , wherein the 2D feature extraction comprises a perspective view (PV) analysis of the images.
3 . The method of claim 1 , wherein the 3D feature extraction comprises a bird's eye view (BEV) analysis of the images.
4 . The method of any claim 1 , wherein the 3D feature extraction is performed by:
generating 3D features from the 2D features resulting from the 2D feature extraction.
5 . The method of claim 4 , wherein the generating the 3D features from the 2D features comprises applying a back-projection model to the 2D features.
6 . The method of claim 4 , wherein a shared pose is further used during the refining.
7 . The method of claim 1 , wherein the 2D feature extraction comprises a 3D object detection method.
8 . An apparatus for detecting objects in images from sensor data, the apparatus comprising at least one processor configured to:
perform, from the sensor data, a 2-dimensional (2D) feature extraction;
perform, from the sensor data, a 3-dimensional (3D) feature extraction;
detect the objects in the images by fusing detection results from the 2D feature extraction and the 3D feature extraction;
refine 3D feature estimates using dual-space object queries that include joint proposals formed based on 2D features resulting from the 2D feature extraction and 3D features resulting from the 3D feature extraction;
wherein the refining comprises performing a multi-level refinement wherein, at each layer of the multi-level refinement, a self-attention layer that acts on both the 2D features and the 3D features, a first cross-attention layer that acts only on the 2D features, and a second cross-attention layer that acts only on the 3D features are used.
9 . The apparatus of claim 8 , wherein the 2D feature extraction comprises a perspective view (PV) analysis of the images and the 3D feature extraction comprises a bird's eye view (BEV) analysis of the images.
10 . The apparatus of claim 8 , wherein the at least one processor performs the 3D feature extraction by:
generating 3D features from the 2D features resulting from the 2D feature extraction.
11 . The apparatus of claim 10 , wherein the generating the 3D features from the 2D features comprises applying a back-projection model to the 2D features.
12 . The apparatus of claim 10 , wherein the refining is performed using a Conv3D or a Conv2D algorithm.
13 . The apparatus of claim 8 , wherein the 2D feature extraction comprises a 3D object detection method.
14 . The apparatus of claim 8 , wherein the 3D feature extraction comprises a dense segmentation and/or a detection method.
15 . A system for deployment on an autonomous vehicle, comprising:
one or more sensors configured to generate sensor data of an environment of the autonomous vehicle; and
at least one processor configured to detect objects in the sensor data by:
performing, using images from the sensor data obtained from the one or more sensors, a 2-dimensional (2D) feature extraction;
performing a 3-dimensional (3D) feature extraction on the sensor data;
detecting objects in the images by fusing detection results from the 2D feature extraction and the 3D feature extraction;
refining 3D feature estimates using dual-space object queries that include joint proposals formed based on 2D features resulting from the 2D feature extraction and 3D features resulting from the 3D feature extraction;
wherein the refining comprises performing a multi-level refinement wherein, at each layer of the multi-level refinement, a self-attention layer that acts on both the 2D features and the 3D features, a first cross-attention layer that acts only on the 2D features, and a second cross-attention layer that acts only on the 3D features are used.
16 . The system of claim 15 , wherein the one or more processor performs the 3D feature extraction by:
generating 3D features from the 2D features resulting from the 2D feature extraction.
17 . The system of claim 15 , wherein hybrid detection proposals are used for querying for the objects.
18 . The system of claim 15 , wherein the one or more sensors include a camera and a lidar.