Method and electronic device for 3D object detection using neural networks
A method of 3D object detection using an object detection neural network includes: receiving one or more monocular images; extracting 2D feature maps from each one of the one or more monocular images by passing the one or more monocular images through a 2D feature extracting part of the object detection neural network, generating an averaged 3D voxel volume based on the 2D feature maps, extracting a 2D representation of 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through an encoder of a 3D feature extracting part of the object detection neural network, and performing 3D object detection as 2D object detection in a Bird's Eye View (BEV) plane, the 2D object detection in the BEV plane being performed by passing the 2D representation of 3D feature maps through thane outdoor object detecting part of the object detection neural network.
1 . A method of three-dimensional (3D) object detection using an object detection neural network comprising a two-dimensional (2D) feature extracting part, a 3D feature extracting part, and an outdoor object detecting part comprising parallel 2D convolutional layers for classification and location, which are pre-trained in end-to-end manner based on posed monocular images, the method comprising:
receiving one or more monocular images;
extracting 2D feature maps from each one of the one or more monocular images by passing the one or more monocular images through the 2D feature extracting part of the object detection neural network,
generating an averaged 3D voxel volume based on the 2D feature maps,
extracting a 2D representation of 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through an encoder of the 3D feature extracting part of the object detection neural network, and
performing 3D object detection as 2D object detection in a Bird's Eye View (BEV) plane, the 2D object detection in the BEV plane being performed by passing the 2D representation of 3D feature maps through the outdoor object detecting part of the object detection neural network.
2 . The method of claim 1 , wherein the method is performed by an electronic device of a vehicle comprising at least one camera configured to detect one or more 3D objects in a surrounding area of the vehicle.
3 . The method of claim 1 , further comprising, for each image of the one or more monocular images, aggregating features in the 2D feature maps corresponding to the respective image via a Feature Pyramid Network (FPN).
4 . The method of claim 1 , wherein the generating the averaged 3D voxel volume further comprises:
for each image of the one or more monocular images, generating a 3D voxel volume corresponding to the respective image using a pinhole camera model that determines a correspondence between 2D coordinates in the corresponding 2D feature maps and 3D coordinates in the 3D voxel volume;
for each of the 3D voxel volume, defining a binary mask corresponding to the respective 3D voxel volume, wherein the binary mask indicates, for each voxel in the 3D voxel volume, whether the respective voxel is inside a camera frustrum of the corresponding image;
for each of the 3D voxel volume, the corresponding binary mask, and the corresponding 2D feature maps, projecting features of the 2D feature maps for each of the voxel inside in the 3D voxel volume as defined by the binary mask;
producing an aggregated binary mask by aggregating the binary masks defined for the 3D voxel volumes of all of the one or more monocular images; and
generating the averaged 3D voxel volume by averaging the features projected in the 3D voxel volumes of all of the one or more monocular images for each of the voxel inside in the 3D voxel volume in the averaged 3D voxel volume as defined by the aggregated binary mask.
5 . The method of claim 1 , wherein the object detection neural network is trained on one or more outdoor training datasets by optimizing a total outdoor loss function based at least on a location loss, a focal loss for classification, and a cross-entropy loss for direction.
6 . A method of three-dimensional (3D) object detection using an object detection neural network comprising a two-dimensional (2D) feature extracting part, a 3D feature extracting part, and an indoor object detecting part comprising 3D convolutional layers for classification, centerness, and location, which are pre-trained in end-to-end manner based on posed monocular images, the method comprising:
receiving one or more monocular images,
extracting 2D feature maps from each one of the one or more monocular images by passing the one or more monocular images through the 2D feature extracting part of the object detection neural network,
generating an averaged 3D voxel volume based on the 2D feature maps,
extracting refined 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through the 3D feature extracting part of the object detection neural network, and
performing 3D object detection by passing the refined 3D feature maps through the indoor object detecting part of the object detection neural network using dense voxel representation of intermediate features.
7 . The method of claim 6 , wherein the method is performed by an electronic device comprising at least one camera configured to detect one or more 3D objects in a surrounding area of the electronic device, and
wherein the electronic device is one of a smartphone, a tablet, smart glasses, or a mobile robot.
8 . The method of claim 6 , further comprising, for each image of the one or more monocular images, aggregating features in the 2D feature maps corresponding to the image via a Feature Pyramid Network (FPN).
9 . The method of claim 6 , wherein the generating the averaged 3D voxel volume further comprises:
for each image of the one or more monocular images, generating a 3D voxel volume corresponding to the respective image using a pinhole camera model that determines a correspondence between 2D coordinates in the corresponding 2D feature maps and 3D coordinates in the 3D voxel volume;
for each of the 3D voxel volume, defining a binary mask corresponding to the respective 3D voxel volume, wherein the binary mask indicates, for each voxel in the 3D voxel volume, whether the respective voxel is inside a camera frustrum of the corresponding image;
for each of the 3D voxel volume, the corresponding binary mask, and the corresponding 2D feature maps, projecting features of the 2D feature maps for each of the voxel inside in the 3D voxel volume as defined by the binary mask;
producing an aggregated binary mask by aggregating the binary masks defined for the 3D voxel volumes of all of the one or more monocular images; and
generating the averaged 3D voxel volume by averaging the features projected in the 3D voxel volumes of all of the one or more monocular images for each of the voxel inside in the 3D voxel volume in the averaged 3D voxel volume as defined by the aggregated binary mask.
10 . The method of claim 6 , wherein the object detection neural network is trained on one or more indoor training datasets by optimizing a total indoor loss function based on at least one of a focal loss for classification, a cross-entropy loss for centerness, or an Intersection-over-Union (IoU) loss for location.
11 . The method of claim 6 , wherein the object detection neural network further comprises a scene understanding part, which is pre-trained in end-to-end manner,
the method further comprising:
performing global average pooling of the extracted 2D feature maps to obtain a tensor representing the extracted 2D feature maps, and
estimating a 1 by passing the tensor through the scene understanding part configured to jointly estimate camera rotation and 3D layout of a scene, and
wherein the scene understanding part comprises two parallel branches: two fully connected layers that output 3D scene layout and two fully other connected layers that estimate camera rotation.
12 . The method of claim 11 , wherein the method is performed by an electronic device to understand the scene in a surrounding area of the electronic device,
wherein the electronic device is one of a smartphone, a tablet, smart glasses, or a mobile robot, and
wherein the electronic device includes at least one camera configured to capture the scene.
13 . The method of claim 11 , wherein the scene understanding part of the object detection neural network is trained on one or more indoor training datasets by optimizing a total scene understanding loss function based on at least one of a layout loss or a camera rotation estimation loss.
14 . An electronic device mounted on a vehicle comprising at least one camera, and configured to use an object detection neural network including a two-dimensional 2D feature extracting part, a 3D feature extracting part, and an outdoor object detecting part including parallel 2D convolutional layers for classification and location, which are pre-trained in end-to-end manner based on posed monocular images, the electronic device comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions to:
receive one or more monocular images;
extract 2D feature maps from each one of the one or more monocular images by passing the one or more monocular images through the 2D feature extracting part of the object detection neural network,
generate an averaged 3D voxel volume based on the 2D feature maps,
extract a 2D representation of 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through an encoder of the 3D feature extracting part of the object detection neural network,
perform 3D object detection as 2D object detection in a Bird's Eye View (BEV) plane, the 2D object detection in the BEV plane being performed by passing the 2D representation of 3D feature maps through the outdoor object detecting part of the object detection neural network; and
detect one or more 3D objects in a surrounding area of the vehicle based on 3D object detection.
15 . An electronic device comprising at least one camera, and configured to use an object detection neural network including a two-dimensional (2D) feature extracting part, a 3D feature extracting part, and an indoor object detecting part including 3D convolutional layers for classification, centerness, and location, which are pre-trained in end-to-end manner based on posed monocular images, the electronic device comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions to:
receive one or more monocular images,
extract 2D feature maps from each one of the one or more monocular images by passing the one or more monocular images through the 2D feature extracting part of the object detection neural network,
generate an averaged 3D voxel volume based on the 2D feature maps,
extract refined 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through the 3D feature extracting part of the object detection neural network,
perform 3D object detection by passing the refined 3D feature maps through the indoor object detecting part of the object detection neural network using dense voxel representation of intermediate features; and
detect one or more 3D objects in a surrounding area of the electronic device based on the 3D object detection.
16 . The electronic device of claim 15 , wherein the electronic device is one of a smartphone, a tablet, smart glasses, or a mobile robot.
17 . A non-transitory computer readable medium storing computer executable instructions that when executed by a processor cause the processor to perform a method of three-dimensional (3D) object detection using an object detection neural network including a two-dimensional 2D feature extracting part, a 3D feature extracting part, and an outdoor object detecting part including parallel 2D convolutional layers for classification and location, which are pre-trained in end-to-end manner based on posed monocular images, the method comprising:
receiving one or more monocular images;
extracting 2D feature maps from each one of the one or more monocular images by passing the one or more monocular images through the 2D feature extracting part of the object detection neural network,
generating an averaged 3D voxel volume based on the 2D feature maps,
extracting a 2D representation of 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through an encoder of the 3D feature extracting part of the object detection neural network, and
performing 3D object detection as 2D object detection in a Bird's Eye View (BEV) plane, the 2D object detection in the BEV plane being performed by passing the 2D representation of 3D feature maps through the outdoor object detecting part of the object detection neural network.