High resolution 3D point clouds generation based on CNN and CRF models
In one embodiment, a method or system generates a high resolution 3-D point cloud to operate an autonomous driving vehicle (ADV) from a low resolution 3-D point cloud and camera-captured image(s). The system receives a first image captured by a camera for a driving environment. The system receives a second image representing a first depth map of a first point cloud corresponding to the driving environment. The system determines a second depth map by applying a convolutional neural network model to the first image. The system generates a third depth map by applying a conditional random fields model to the first image, the second image and the second depth map, the third depth map having a higher resolution than the first depth map such that the third depth map represents a second point cloud perceiving the driving environment surrounding the ADV.
1. A computer-implemented method for operating an autonomous driving vehicle (ADV), the method comprising:
receiving a first image captured by a first camera, the first image capturing a portion of a driving environment of the ADV;
receiving a second image representing a first depth map of a first point cloud corresponding to the portion of the driving environment produced by a light detection and ranging (LiDAR) device;
determining a second depth map by applying a convolutional neural network (CNN) model to the first image;
generating a third depth map by applying a conditional random fields (CRF) model to the first image, the second image, and the second depth map, including optimizing, using the CRF model, an overall cost in generating the third depth map, wherein the overall cost comprises at least an image matching cost for estimating image pixel smoothness, and a smoothness pairwise term for estimating image pixel discontinuities, wherein the third depth map has a higher resolution than the first depth map; and
operating the ADV using the third depth map, which represents a second point cloud utilized to perceive the driving environment surrounding the ADV.
2. The method of claim 1 , further comprising:
receiving a third image captured by a second camera; and
determining the second depth map by applying the CNN model to the first and the third images.
3. The method of claim 1 , wherein the first image comprises a cylindrical panorama image or a spherical panorama image.
4. The method of claim 3 , wherein the cylindrical panorama image or the spherical panorama image is generated based on a plurality of images captured by a plurality of camera devices.
5. The method of claim 3 , further comprising reconstructing the second point cloud by projecting the second depth map into a 3-D space based on the cylindrical panorama image or the spherical panorama image.
6. The method of claim 1 , further comprising mapping the third image onto an image plane of the first image.
7. The method of claim 6 , wherein the third depth map is generated by blending one or more generated depth maps, wherein the third depth map is a panorama map.
8. The method of claim 1 , wherein the CNN model comprises:
a plurality of contractive layers, wherein each contractive layer includes an encoder to downsample a respective input; and
a plurality of expansive layers coupled to the plurality of contractive layers, wherein each expansive layer includes a decoder to upsample a respective input.
9. The method of claim 8 , wherein information of the plurality of contractive layers are fed forward to the plurality of expansive layers.
10. The method of claim 8 , wherein each of the plurality of expansive layers includes a prediction layer to predict a depth map for a subsequent layer.
11. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving a first image captured by a first camera, the first image capturing a portion of a driving environment of an autonomous driving vehicle (ADV);
receiving a second image representing a first depth map of a first point cloud corresponding to the portion of the driving environment produced by a light detection and ranging (LiDAR) device;
determining a second depth map by applying a convolutional neural network (CNN) model to the first image; and
generating a third depth map by applying a conditional random fields (CRF) model to the first image, the second image, and the second depth map, including optimizing, using the CRF model, an overall cost in generating the third depth map, wherein the overall cost comprises at least an image matching cost for estimating image pixel smoothness, and a smoothness pairwise term for estimating image pixel discontinuities, wherein the third depth map has a higher resolution than the first depth map; and
operating the ADV using the third depth map, which represents a second point cloud utilized to perceive the driving environment surrounding the ADV.
12. The non-transitory machine-readable medium of claim 11 , further comprising:
receiving a third image captured by a second camera; and
determining the second depth map by applying the CNN model to the first and the third images.
13. The non-transitory machine-readable medium of claim 11 , wherein the first image comprises a cylindrical panorama image or a spherical panorama image.
14. The non-transitory machine-readable medium of claim 13 , wherein the cylindrical panorama image or the spherical panorama image is generated based on a plurality of images captured by a plurality of camera devices.
15. The non-transitory machine-readable medium of claim 13 , further comprising reconstructing the second point cloud by projecting the second depth map into a 3-D space based on the cylindrical panorama image or the spherical panorama image.
16. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
receiving a first image captured by a first camera, the first image capturing a portion of a driving environment of an autonomous driving vehicle (ADV);
receiving a second image representing a first depth map of a first point cloud corresponding to the portion of the driving environment produced by a light detection and ranging (LiDAR) device;
determining a second depth map by applying a convolutional neural network (CNN) model to the first image; and
generating a third depth map by applying a conditional random fields (CRF) model to the first image, the second image, and the second depth map, including optimizing, using the CRF model, an overall cost in generating the third depth map, wherein the overall cost comprises at least an image matching cost for estimating image pixel smoothness, and a smoothness pairwise term for estimating image pixel discontinuities, wherein the third depth map has a higher resolution than the first depth map; and
operating the ADV using the third depth map, which represents a second point cloud utilized to perceive the driving environment surrounding the ADV.
17. The system of claim 16 , further comprising:
receiving a third image captured by a second camera; and
determining the second depth map by applying the CNN model to the first and the third images.
18. The system of claim 16 , wherein the first image comprises a cylindrical panorama image or a spherical panorama image.
19. The system of claim 18 , wherein the cylindrical panorama image or the spherical panorama image is generated based on a plurality of images captured by a plurality of camera devices.
20. The system of claim 18 , further comprising reconstructing the second point cloud by projecting the second depth map into a 3-D space based on the cylindrical panorama image or the spherical panorama image.