Fusion of RGB images and LiDAR data for lane classification

1. A method comprising:

receiving point cloud data;

segmenting a ground plane from the point cloud data;

generating a two-dimensional image of the segmented ground plane based on intensity values of the segmented ground plane;

determining lane marking candidates based on intensity within the generated two-dimensional image;

receiving image data;

registering the generated two-dimensional image with the received image data;

determining lane marking candidates of the received image data based on the determined lane marking candidates of the registered two-dimensional image; and

selecting a plurality of image patches from the registered two-dimensional image and from the registered image data based on the determined lane markings.

2. The method of claim 1 , further comprising:

generating a plurality of feature maps from the plurality of image patches wherein the plurality of image patches includes selected image patches from the registered two-dimensional image and selected image patches from the received image data;

sub-sampling the plurality of feature maps;

generating a feature vector based on the plurality of feature maps; and

determining lane markings from the generated feature vector based on intensity.

3. The method of claim 1 , wherein determining lane marking candidates within the further comprises:

applying a positive threshold to determine lane marking candidates; and

applying a negative threshold to determine negative lane marking candidates, wherein the positive threshold is higher than the negative threshold.

4. The method of claim 1 , wherein registering the generated two-dimensional image with the received image data is based on lane markings.

5. The method of claim 1 , further comprising:

increasing the contrast of the generated two-dimensional image based on intensity values of the segmented point cloud data.

6. The method of claim 1 , wherein segmenting the ground plane further comprises one of thresholding based on height, the normal of a plurality of points of the three-dimensional point cloud, or plane fitting.

7. The method of claim 1 , further comprising

smoothing gaps between projected points on the generated two-dimensional image.

8. The method of claim 1 , wherein determining lane marking candidates based on intensity within the generated two-dimensional image comprises:

computing the convex hull of white pixels of the generated two-dimensional image.

9. The method of claim 2 , wherein generating the plurality of feature maps further comprises:

determining the classification of each pixel based on a sliding window.

10. The method of claim 1 , wherein the received image data comprises at least one color image.

11. The method of claim 10 , further comprising;

transforming a color space of the received image data.

12. The method of claim 1 , wherein the received point cloud data and the received image data include geoposition information and pose information.

13. The method of claim 1 , wherein the point cloud data and the image data are collected simultaneously.

14. An apparatus comprising:

at least one processor; and

at least one memory including computer program code for one or more programs; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:

receive point cloud data;

segment a ground plane from the point cloud data;

generate a two-dimensional image of the segmented ground plane based on intensity values of the segmented ground plane;

determine lane marking candidates based on intensity within the two-dimensional image;

receive image data;

register the generated two-dimensional image with the received image data;

determine lane marking candidates of the received image data based on the determined lane marking candidates of the registered two-dimensional image;

select a plurality of image patches from the two-dimensional image and from the received image data based on the determined lane markings;

generate a plurality of feature maps from the plurality of image patches wherein the plurality of image patches includes selected image patches from the registered two-dimensional image and selected image patches from the received image data;

sub-sample the set of feature maps;

generate a feature vector based on the set of feature maps; and

determine lane markings from the generated feature vector based on intensity.

15. The apparatus of claim of claim 14 , the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:

apply a positive threshold to determine the presence of lane marking candidates; and

apply a negative threshold to determine an absence of lane marking candidates, wherein the positive threshold is higher than the negative threshold.

16. The apparatus of claim 14 , the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:

increase the contrast of the generated two-dimensional image based on intensity values of the segmented point cloud data.

17. The apparatus of claim 14 , the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:

smooth gaps between projected points on the generated two-dimensional image.

18. A non-transitory computer readable medium including instructions that when executed are operable to:

receive a plurality of color images;

receive point cloud data;

generate a plurality of two-dimensional images based on intensity of the point cloud data;

register each of the generated two-dimensional images with color images of the plurality of color images based on geolocation and pose;

generate a plurality of feature maps from each registered two-dimensional image and its corresponding color image based on a classifier; wherein the classifier is based on positive lane marking examples and negative lane marking examples;

sub-sample the plurality of feature maps;

generate a feature vector based on the plurality of feature maps; and

determine lane markings from the generated feature vector based on intensity.

19. The non-transitory computer readable medium of claim 18 , including instructions that when executed are operable to:

smooth gaps between projected points on the determined lane markings.

20. The non-transitory computer readable medium of claim 18 , including instructions that when executed are operable to:

apply the classifier to each pixel of the of each registered two-dimensional image and its corresponding color image based on a sliding window.