Automatic lane marking extraction and classification from lidar scans
Provided are methods, systems, and computer program products for generating an output map indicating a likelihood of individual elements of an image as corresponding to particular road elements, such as lane dividers, road dividers, and road boundaries. An example method may include applying a machine learning architecture to the image, which architecture includes a convolutional neural network and a sub-network capturing global context from feature maps generated by the convolutional neural network.
1. A computer-implemented method implemented by at least one processor, the computer-implemented method comprising:
receiving, by the at least one processor, an image representing data from a liDAR scan of a vehicle environment;
convoluting, by the at least one processor, the image to generate a first feature map;
converting, by the at least one processor, the first feature map into a first feature vector;
applying, by the at least one processor, one or more additional convolutions to the first feature map to generate a second feature map;
converting, by the at least one processor, the second feature map into a second feature vector;
passing, by the at least one processor, an input representing at least the first feature vector and the second feature vector through a neural network to produce a scene feature vector;
generating, by the at least one processor, an output map based on at least the scene feature vector and the second feature map, wherein the output map includes a plurality of pixels and indicates a likelihood of each pixel corresponding to a road element; and
generating one or more polylines from the output map, each of the one or more polylines identifying a road element in the vehicle environment, wherein generating the one or more polylines comprises:
converting one or more adjacent pixels in the output map with a classification value satisfying a threshold into an undirected graph; and
identifying a longest path within the undirected graph as a polyline of the one or more polylines.
2. The computer-implemented method of claim 1 , wherein the image is a two-dimensional pseudo-image generated from a three-dimensional liDAR point cloud image.
3. The computer-implemented method of claim 2 , wherein the three-dimensional liDAR point cloud image is generated from an aggregate three-dimensional liDAR point cloud image representing multiple liDAR scans of the vehicle environment.
4. The computer-implemented method of claim 1 , wherein generating the output map comprises generating a segmentation map that associates a classification value to each pixel of the plurality of pixels, the classification value indicating the likelihood that the pixel corresponds to the road element.
5. The computer-implemented method of claim 4 , wherein the road element is one of a plurality of road elements, and wherein the output map associates multiple classification values to each pixel, each of the multiple classification values corresponding to a respective road element of the plurality of road elements.
6. The computer-implemented method of claim 5 , wherein the plurality of road elements comprises at least one of a lane divider indicating a division between lanes of traffic in a same direction, a road divider indicating a division between lanes of traffic in different directions, a road boundary element indicating a boundary of a roadway, stop lines, pedestrian markings, bike markings, and chevron markings.
7. The computer-implemented method of claim 1 , wherein applying one or more additional convolutions to the first feature map to generate the second feature map comprises applying a first additional convolution to the first feature map to generate an intermediary feature map and applying a second additional convolution to the intermediary feature map to generate the second feature map.
8. The computer-implemented method of claim 1 , wherein converting the first feature map into the first feature vector comprises reducing a dimensionality of the first feature map using a pooling operation.
9. The computer-implemented method of claim 1 , wherein the input representing at least the first feature vector and the second feature vector is a concatenation of at least the first feature vector and the second feature vectors.
10. The computer-implemented method of claim 1 , wherein generating the output map based on the scene feature vector and the second feature map includes deconvolving the second feature map based on at least the scene feature vector.
11. The computer-implemented method of claim 1 , further comprising generating one or more polylines from the output map, each of the one or more polylines identifying a road element in the vehicle environment.
12. A system comprising:
one or more non-transitory data stores including computer-executable instructions; and
one or more hardware processors configured to execute the computer-executable instructions to:
receive an image representing data from a liDAR scan of a vehicle environment;
convolute the image to generate a first feature map;
convert the first feature map into a first feature vector;
apply one or more additional convolutions to the first feature map to generate a second feature map;
convert the second feature map into a second feature vector;
pass an input representing at least the first feature vector and the second feature vector through a neural network to produce a scene feature vector; and
generate an output map based on at least the scene feature vector and the second feature map, wherein the output map includes a plurality of pixels and indicates a likelihood of each pixel corresponding to a road element; and
generate one or more polylines from the output map, each of the one or more polylines identifying a road element in the vehicle environment, wherein to generate the one or more polylines, the computer-executable instructions cause the system to:
convert one or more adjacent pixels in the output map with a classification value satisfying a threshold into an undirected graph; and
identify a longest path within the undirected graph as a polyline of the one or more polylines.
13. The system of claim 12 , wherein to generate the one or more polylines, the computer-executable instructions, when executed by the one or more hardware processors, further cause the system to skeletonize the output map.