Multimodal weakly-supervised three-dimensional (3D) object detection method and system, and device
Disclosed are a multimodal weakly-supervised three-dimensional (3D) object detection method and system, and a device. The method includes: shooting multiple two-dimensional (2D) red, green and blue (RGB) images with a camera, acquiring ground points by a vehicle LiDAR sensor and generating a 3D frustum based on 2D box labels on each of the 2D RGB images; filtering ground points in the 3D frustum and selecting a region with most 3D points; generating a 3D pseudo-labeling bounding box of an object according to the region with the most 3D points; training a multimodal superpixel dual-branch network with the 3D pseudo-labeling bounding boxes as labels and the 2D RGB image and the 3D point cloud as inputs; and inputting a 2D RGB image of a current frame and a 3D point cloud of a current scenario to a trained multimodal superpixel dual-branch network to generate an overall 3D point cloud.
1 . A multimodal weakly-supervised three-dimensional (3D) object detection method, comprising:
shooting multiple two-dimensional (2D) red, green and blue (RGB) images with a camera, acquiring 3D points by a vehicle LiDAR sensor, and generating a 3D frustum based on 2D box labels on each of the 2D RGB images;
filtering ground points in the 3D frustum and selecting a region with most 3D points in a filtered 3D frustum based on seeded region growing (SRG);
generating a 3D pseudo-labeling bounding box of an object according to the region with the most 3D points in the filtered 3D frustum based on density-based 3D parameter estimation, which is operated in a main control chip, the object being a pedestrian or a cyclist;
training a multimodal superpixel dual-branch network with the 3D pseudo-labeling bounding boxes as labels and the 2D RGB image and 3D point cloud as inputs in the main control chip; and
acquiring a 2D RGB image of a current frame shot by the camera and a 3D point cloud of a current scenario acquired by the vehicle LiDAR sensor, and inputting the 2D RGB image of the current frame and the 3D point cloud of the current scenario to the trained multimodal superpixel dual-branch network to generate a 3D object prediction bounding box of the current scenario, wherein the 3D object prediction bounding box is configured to localize the object around a vehicle, and predict a size and an orientation of the object around the current vehicle;
wherein the training a multimodal superpixel dual-branch network with the 3D pseudo-labeling bounding boxes as labels and 2D RGB image and the 3D point cloud as inputs for generating a trained multimodal superpixel dual-branch network specifically comprises:
corresponding 2D local pixel blocks to 3D points spatially with a local consistency priori to generate a projection relationship;
extracting an image feature map of the 2D RGB image with ResNet50, and upsampling the image feature map to the same size of an input image, the input image being the 2D RGB image;
performing superpixel segmentation on the input image, and obtaining superpixel features by pooling the image feature map according to a superpixel segmentation result;
assigning the superpixel features to the 3D points according to the projection relationship;
applying separately 3D convolution for 3D point cloud with initial geometric features and 3D point cloud with superpixel features, and projecting the point cloud features to BEV generating a BEV feature map;
fusing 3D point cloud features generated by the initial geometric features and superpixel features in the BEV feature map to generate a fused feature; and
training the multimodal superpixel dual-branch network with the fused feature as an object classification feature to generate the trained multimodal superpixel dual-branch network.
2 . The multimodal weakly-supervised 3D object detection method according to claim 1 , wherein the generating a 3D pseudo-labeling bounding box of an object according to the region with the most 3D points in the filtered 3D frustum based on density-based 3D parameter estimation, which is operated in a main control chip, specifically comprises:
filtering an adhesive noise point cloud in the region with the most 3D points to generate a filtered 3D point cloud;
iteratively searching a bird's eye view (BEV) rotating box of the filtered 3D point cloud, progressively deleting a point with minimum density in the filtered 3D point cloud, and exiting iteration once a position of a key point is kept unchanged, thereby generating the BEV rotating box of the filtered 3D point cloud, wherein the key point is an angular point comprising most 3D points among four angular points of the rotating box; and
expanding the key point to two sides of the filtered 3D frustum, constricting an expansion length with statistical information of the object, and generating the 3D pseudo-labeling bounding box with a vector direction of a long side as an orientation of the object.
3 . The multimodal weakly-supervised 3D object detection method according to claim 1 , further comprising:
training the multimodal superpixel dual-branch network by randomly abandoning the superpixel features, and padding abandoned superpixel features with 0.
4 . A multimodal weakly-supervised three-dimensional (3D) object detection system, comprising:
a 3D frustum generation module configured to shoot multiple two-dimensional (2D) red, green and blue (RGB) images with a camera, acquire 3D point cloud by a vehicle LiDAR sensor, and generate a 3D frustum based on 2D box labels on each of the 2D RGB images;
a filtration module configured to filter ground points in the 3D frustum, and select a region with most 3D points in a filtered 3D frustum based on seeded region growing (SRG);
a 3D pseudo-labeling bounding box generation module configured to generate a 3D pseudo-labeling bounding box of an object according to the region with the most 3D points in the filtered 3D frustum based on density-based 3D parameter estimation, which is operated in a main control chip, the object being a pedestrian or a cyclist;
a training module configured to train a multimodal superpixel dual-branch network with the 3D pseudo-labeling bounding boxes as labels and the 2D RGB image and the 3D point cloud as inputs in the main control chip; and
a detection module configured to acquire a 2D RGB image of a current frame shot by the camera and a 3D point cloud of a current scenario acquired by the vehicle LiDAR sensor, and input the 2D RGB image of the current frame and the 3D point cloud of the current scenario to the trained multimodal superpixel dual-branch network to generate a 3D object prediction bounding box of the current scenario, wherein the 3D object prediction bounding box is configured to localize the object around a vehicle, and predict a size and an orientation of the object around the current vehicle;
wherein the training module specifically comprises:
a projection relationship generation unit configured to correspond 2D local pixel blocks to 3D points spatially with a local consistency priori to generate a projection relationship;
an upsampling unit configured to extract an image feature map of the 2D RGB image with ResNet50, and upsample the image feature map to the same size of an input image, the input image being the 2D RGB image;
a superpixel feature obtaining unit configured to perform superpixel segmentation on the input image, and obtain superpixel features by pooling on the image feature map according to a superpixel segmentation result;
an assignment unit configured to assign the superpixel features to the 3D points according to the projection relationship;
a projection unit configured to apply separately 3D convolution for 3D point cloud with initial geometric features and 3D point cloud with superpixel features, and project the 3D point cloud features to BEV generating a BEV feature map;
a fused feature generation unit configured to fuse point cloud features generated by initial geometric features and superpixel features in the BEV feature map to generate a fused feature; and
a trained multimodal superpixel dual-branch network generation unit configured to train the multimodal superpixel dual-branch network with the fused feature as an object classification feature to generate the trained multimodal superpixel dual-branch network.
5 . The multimodal weakly-supervised 3D object detection system according to claim 4 , wherein the 3D pseudo-labeling bounding box generation module specifically comprises:
a filtration unit configured to filter an adhesive noise point cloud in the region with the most 3D points to generate a filtered 3D point cloud;
a bird's eye view (BEV) rotating box generation unit configured to iteratively search a BEV rotating box of the filtered 3D point cloud, progressively delete a point with minimum density in the filtered 3D point cloud, and exit iteration once a position of a key point is kept unchanged, thereby generating the BEV rotating box of the filtered 3D point cloud, wherein the key point is an angular point comprising most 3D points among four angular points of the rotating box; and
a 3D pseudo-labeling bounding box generation unit configured to expand the key point to two sides of the filtered 3D frustum, constrict an expansion length with statistical information of the object, and generate the 3D pseudo-labeling bounding box with a vector direction of a long side as an orientation of the object.
6 . The multimodal weakly-supervised 3D object detection system according to claim 4 , further comprising:
a random superpixel feature abandonment module configured to train the multimodal superpixel dual-branch network by randomly abandoning the superpixel features, and pad abandoned superpixel features with 0.
7 . An electronic device, comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor runs the computer program to enable the electronic device to execute the multimodal weakly-supervised three-dimensional (3D) object detection method according to claim 1 .
8 . The electronic device according to claim 7 , wherein the generating a 3D pseudo-labeling bounding box of an object according to the region with the most 3D points in the filtered 3D frustum based on density-based 3D parameter estimation, which is operated in a main control chip, specifically comprises:
filtering an adhesive noise point cloud in the region with the most 3D points to generate a filtered 3D point cloud;
iteratively searching a bird's eye view (BEV) rotating box of the filtered 3D point cloud, progressively deleting a point with minimum density in the filtered 3D point cloud, and exiting iteration once a position of a key point is kept unchanged, thereby generating the BEV rotating box of the filtered 3D point cloud, wherein the key point is an angular point comprising most 3D points among four angular points of the rotating box; and
expanding the key point to two sides of the filtered 3D frustum, constricting an expansion length with statistical information of the object, and generating the 3D pseudo-labeling bounding box with a vector direction of a long side as an orientation of the object.
9 . The electronic device according to claim 7 , further comprising:
training the multimodal superpixel dual-branch network by randomly abandoning the superpixel features, and padding abandoned superpixel features with 0.