System and method for large-scale lane marking detection using multimodal sensor data
A system and method for large-scale lane marking detection using multimodal sensor data are disclosed. A particular embodiment includes: receiving image data from an image generating device mounted on a vehicle; receiving point cloud data from a distance and intensity measuring device mounted on the vehicle; fusing the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data; and generating a lane marking map from the set of lane marking points.
1. A system comprising:
a data processor; and
a multimodal lane detection module, executable by the data processor, the multimodal lane detection module being configured to perform a multimodal lane detection operation configured to:
receive image data from an image generating device mounted on a vehicle, the received image data corresponding to a particular location;
receive point cloud data from a distance and intensity measuring device mounted on the vehicle;
fuse the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data, the fusion including aligning and orienting the image data with a terrain map corresponding to the particular location and using terrain map elevation data to transform the image data to the 3D space; and
generate a lane marking map from the set of lane marking points.
2. The system of claim 1 being configured to perform a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.
3. The system of claim 2 being configured to train a neural network to perform the semantic segmentation operation.
4. The system of claim 1 wherein the image generating device is one or more cameras.
5. The system of claim 1 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.
6. The system of claim 1 being configured to receive vehicle metrics from a vehicle subsystem.
7. The system of claim 1 being configured to back-project the image data on the terrain map with the terrain map elevation data.
8. The system of claim 1 being further configured to output the lane marking map to a vehicle control subsystem of the vehicle.
9. A method comprising:
receiving image data from an image generating device mounted on a vehicle, the received image data corresponding to a particular location;
receiving point cloud data from a distance and intensity measuring device mounted on the vehicle;
fusing the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data the fusing including aligning and orienting the image data with a terrain map corresponding to the particular location and using terrain map elevation data to transform the image data to the 3D space; and
generating a lane marking map from the set of lane marking points.
10. The method of claim 9 including performing a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.
11. The method of claim 10 including training a neural network to perform the semantic segmentation operation.
12. The method of claim 9 wherein the image generating device is one or more cameras.
13. The method of claim 9 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.
14. The method of claim 9 including receiving vehicle metrics from a vehicle subsystem.
15. The method of claim 9 including back-projecting the image data on the terrain map with the terrain map elevation data.
16. The method of claim 9 including outputting the lane marking map to a vehicle control subsystem of the vehicle.
17. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:
receive image data from an image generating device mounted on a vehicle, the received image data corresponding to a particular location;
receive point cloud data from a distance and intensity measuring device mounted on the vehicle;
fuse the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data, the fusion including aligning and orienting the image data with a terrain map corresponding to the particular location and using terrain map elevation data to transform the image data to the 3D space; and
generate a lane marking map from the set of lane marking points.
18. The non-transitory machine-useable storage medium of claim 17 being configured to perform a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.
19. The non-transitory machine-useable storage medium of claim 17 being further configured to fit piecewise lines for each lane marking object detected in the received image data.
20. The non-transitory machine-useable storage medium of claim 17 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.