System and method for drivable road surface representation generation using multimodal sensor data

1. A system comprising:

a data processor; and

a multimodal drivable road surface detection module, executable by the data processor, the multimodal drivable road surface detection module being configured to perform a multimodal drivable road surface detection operation configured to:

receive image data from an image generating device mounted on a vehicle and to receive three dimensional (3D) point cloud data from a distance measuring device mounted on the vehicle;

project the 3D point cloud data onto the 2D image data to produce mapped image and point cloud data;

perform post-processing operations on the mapped image and point cloud data, the post-processing operations being configured to perform an outlier detection process to remove outlier points of the point cloud data that do not correspond to a flat road surface captured in the image data, the post-processing operations being further configured to perform a density-based spatial clustering process to remove outlier points of the point cloud data that do not correspond to a point cluster; and

perform a smoothing operation on the processed mapped image and point cloud data to produce a drivable road surface map or representation.

2. The system of claim 1 being configured to perform a primary filtering operation on the 3D point cloud data using a Random Sample Consensus (RANSAC) operation.

3. The system of claim 1 being configured to perform a secondary filtering and mapping operation on the image data and the 3D point cloud data using a density-based spatial clustering of applications with noise (DBSCAN) operation.

4. The system of claim 1 being configured to perform pixel dilation and post-processing operations on the mapped image and point cloud data.

5. The system of claim 1 being configured to use the drivable road surface map or representation to train a deep convolutional neural network (CNN).

6. The system of claim 1 wherein the image generating device is one or more cameras.

7. The system of claim 1 wherein the distance measuring device is one or more laser light detection and ranging (LIDAR) devices.

8. The system of claim 1 being further configured to output the drivable road surface map to a vehicle control subsystem of the vehicle.

9. A method comprising:

receiving image data from an image generating device mounted on a vehicle and receiving three dimensional (3D) point cloud data from a distance measuring device mounted on the vehicle;

projecting the 3D point cloud data onto the 2D image data to produce mapped image and point cloud data;

performing post-processing operations on the mapped image and point cloud data, the post-processing operations including performing an outlier detection process to remove outlier points of the point cloud data that do not correspond to a flat road surface captured in the image data, the post-processing operations including performing a density-based spatial clustering process to remove outlier points of the point cloud data that do not correspond to a point cluster; and

performing a smoothing operation on the processed mapped image and point cloud data to produce a drivable road surface map or representation.

10. The method of claim 9 including performing a primary filtering operation on the 3D point cloud data using a Random Sample Consensus (RANSAC) operation.

11. The method of claim 9 including performing a secondary filtering and mapping operation on the image data and the 3D point cloud data using a density-based spatial clustering of applications with noise (DBSCAN) operation.

12. The method of claim 9 including performing pixel dilation and post-processing operations on the mapped image and point cloud data.

13. The method of claim 9 including using the drivable road surface map or representation to train a deep convolutional neural network (CNN).

14. The method of claim 9 wherein the image generating device is one or more cameras.

15. The method of claim 9 wherein the distance measuring device is one or more laser light detection and ranging (LIDAR) devices.

16. The method of claim 9 including outputting the drivable road surface map to a vehicle control subsystem of the vehicle.

17. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive image data from an image generating device mounted on a vehicle and to receive three dimensional (3D) point cloud data from a distance measuring device mounted on the vehicle;

project the 3D point cloud data onto the 2D image data to produce mapped image and point cloud data;

perform post-processing operations on the mapped image and point cloud data, the post-processing operations being configured to perform an outlier detection process to remove outlier points of the point cloud data that do not correspond to a flat road surface captured in the image data, the post-processing operations being further configured to perform a density-based spatial clustering process to remove outlier points of the point cloud data that do not correspond to a point cluster; and

perform a smoothing operation on the processed mapped image and point cloud data to produce a drivable road surface map or representation.

18. The non-transitory machine-useable storage medium of claim 17 being configured to perform a primary filtering operation on the 3D point cloud data using a Random Sample Consensus (RANSAC) operation.

19. The non-transitory machine-useable storage medium of claim 17 being configured to perform a secondary filtering and mapping operation on the image data and the 3D point cloud data using a density-based spatial clustering of applications with noise (DBSCAN) operation.

20. The non-transitory machine-useable storage medium of claim 17 being configured to perform pixel dilation and post-processing operations on the mapped image and point cloud data.