Deep learning-based feature extraction for LiDAR localization of autonomous driving vehicles

1. A computer-implemented method for localizing an autonomous driving vehicle (ADV), the computer-implemented method comprising:

selecting a first set of keypoints from an online point cloud, the online point cloud generated by a light detection and ranging (LiDAR) device on the ADV;

extracting a first set of feature descriptors for the first set of keypoints using a feature learning neural network running on the ADV;

locating a second set of keypoints on a pre-built point cloud map, each keypoint of the second set of keypoints corresponding to a keypoint of the first set of keypoints;

extracting a second set of feature descriptors from the pre-built point cloud map;

constructing a cost volume from the first set of feature descriptors and the second set of feature descriptors, where the cost volume includes a plurality of cells, each cell representing a matching cost between one of the first set of keypoints and a corresponding keypoint of the second set of keypoints;

regularizing the cost volume using a plurality of different types of neural networks to generate additional contexts for refining the matching cost in the cost volume;

and

estimating a position and an orientation of the ADV by minimizing the matching cost in the cost volume.

2. The computer-implemented method of claim 1 , wherein the matching cost is associated with a given offset.

3. The computer-implemented method of claim 1 , wherein each of the first set of keypoints and the second set of keypoints is a LiDAR point, and has a fixed number of keypoints.

4. The computer-implemented method of claim 3 , wherein selecting the first set of keypoints further comprises:

traversing LiDAR points in a predetermined area around the predicted pose of the ADV to locate a plurality of candidate LiDAR points, each candidate LiDAR point having a predetermined density of LiDAR points in a neighborhood of that candidate LiDAR point;

evaluating a linearity of each of the plurality of candidate LiDAR points using a 3D structure tensor to assign a value to the linearity of the candidate LiDAR point;

sorting the plurality of candidate LiDAR points based on their assigned values; and

selecting a fixed number of candidate LiDAR points based on the sorted candidate LiDAR points.

5. The computer-implemented method of claim 4 , wherein each newly selected LiDAR point of the fixed number of candidate LiDAR points is to maintain a predetermined distance from one or more existing LiDAR points that have been selected.

6. The computer-implemented method of claim 1 , wherein the feature learning neural network comprises a plurality of mini-neural networks, each mini-neural network provided for one of the first set of keypoints, wherein each mini-neural neural network is to

apply a cylinder along a z-axis of the keypoint to collect a predetermined number of neighboring LiDAR points, each neighboring LiDAR point having a reflection intensity and a relative coordinate to that keypoint; and

extract a feature descriptor for the keypoint based on the relative coordinate and reflection intensities of the predetermined number of neighboring LiDAR points.

7. The computer-implemented method of claim 1 , wherein the feature learning neural network includes a plurality of mini-neural networks, each mini-neural network provided for one of the second set of keypoints, wherein each mini-neural neural network is to

apply a cylinder along a z-axis of the keypoint to collect a predetermined number of neighboring LiDAR points, each neighboring LiDAR point having a reflection intensity and a relative coordinate to that keypoint; and

extract a feature descriptor for the keypoint based on the relative coordinates

and reflection intensities of the predetermined number of neighboring LiDAR points.

8. The computer-implemented method of claim 7 , wherein when the predetermined number of neighboring LiDAR points are not found for a keypoint in the second set of keypoints, a fully connected network is to extract a feature descriptor for the keypoint.

9. The computer-implemented method of claim 1 , wherein coordinates for each of the second set of keypoints on the pre-built point cloud map are to be computed using a transform expressed by a 2 by 2 rotation matrix and 2D translation vector.

10. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor of an autonomous driving vehicle (ADV), cause the ADV to perform operations, the operations comprising:

selecting a first set of keypoints from an online point cloud, the online point cloud generated by a light detection and ranging (LiDAR) device on the ADV;

extracting a first set of feature descriptors for the first set of keypoints using a feature learning neural network running on the ADV;

locating a second set of keypoints on a pre-built point cloud map, each keypoint of the second set of keypoints corresponding to a keypoint of the first set of keypoints;

extracting a second set of feature descriptors from the pre-built point cloud map;

constructing a cost volume from the first set of feature descriptors and the second set of feature descriptors, where the cost volume includes a plurality of cells, each cell representing a matching cost between one of the first set of keypoints and a corresponding keypoint of the second set of keypoints;

regularizing the cost volume using a plurality of different types of neural networks to generate additional contexts for refining the matching cost in the cost volume;

and

estimating a position and an orientation of the ADV by minimizing the matching cost in the cost volume.

11. The non-transitory machine-readable medium of claim 10 , wherein the matching cost is associated with a given offset.

12. The non-transitory machine-readable medium of claim 10 , wherein each of the first set of keypoints and the second set of keypoints is a LiDAR point, and has a fixed number of keypoints.

13. The non-transitory machine-readable medium of claim 12 , wherein selecting the first set of keypoints further comprises:

traversing LiDAR points in a predetermined area around the predicted pose of the ADV to locate a plurality of candidate LiDAR points, each candidate LiDAR point having a predetermined density of LiDAR points in a neighborhood of that candidate LiDAR point;

evaluating a linearity of each of the plurality of candidate LiDAR points using a 3D structure tensor to assign a value to the linearity of each of the plurality of candidate LiDAR points;

sorting the plurality of candidate LiDAR points based on their assigned values; and

selecting a fixed number of candidate LiDAR points based on the sorted candidate LiDAR points.

14. The non-transitory machine-readable medium of claim 13 , wherein each newly selected LiDAR point of the fixed number of candidate LiDAR points is to maintain a predetermined distance from one or more existing LiDAR points that have been selected.

15. The non-transitory machine-readable medium of claim 10 , wherein the feature learning neural network comprises a plurality of mini-neural networks, each mini-neural network provided for one of the first set of keypoints, wherein each mini-neural neural network is to

apply a cylinder along a z-axis of the keypoint to collect a predetermined number of neighboring LiDAR points, each neighboring LiDAR point having a reflection intensity and a relative coordinate to that keypoint; and

extract a feature descriptor for the keypoint based on the relative coordinate and

reflection intensities of the predetermined number of neighboring LiDAR points.

16. The non-transitory machine-readable medium of claim 10 , wherein the feature learning neural network includes a plurality of mini-neural networks, each mini-neural network provided for one of the second set of keypoints, wherein each mini-neural neural network is to

apply a cylinder along a z-axis of the keypoint to collect a predetermined number of neighboring LiDAR points, each neighboring LiDAR point having a reflection intensity and a relative coordinate to that keypoint; and

extracting a feature descriptor for the keypoint based on the relative coordinate and reflection intensities of the predetermined number of neighboring LiDAR points.

17. The non-transitory machine-readable medium of claim 16 , wherein when the predetermined number of neighboring LiDAR points are not found for a keypoint in the second set of keypoints, a fully connected network is to extract a feature descriptor for the keypoint.

18. The non-transitory machine-readable medium of claim 10 , wherein coordinates for each of the second set of keypoints on the pre-built point cloud map are to be computed using a transform expressed by a 2 by 2 rotation matrix and 2D translation vector.

19. A system for localizing an autonomous driving vehicle (ADV), the system comprising:

a processor; and

a memory coupled to the processor to store instructions, which when executed by the processor, cause the ADV to perform operations, the operations including;

selecting a first set of keypoints from an online point cloud, the online point cloud generated by a light detection and ranging (LiDAR) device on the ADV,

extracting a first set of feature descriptors for the first set of keypoints using a feature learning neural network running on the ADV,

locating a second set of keypoints on a pre-built point cloud map, each keypoint of the second set of keypoints corresponding to a keypoint of the first set of keypoints,

extracting a second set of feature descriptors from the pre-built point cloud map,

constructing a cost volume from the first set of feature descriptors and the second set of feature descriptors, where the cost volume includes a plurality of cells, each cell representing a matching cost between one of the first set of keypoints and a corresponding keypoint of the second set of keypoints,

regularizing the cost volume using a plurality of different types of neural networks to generate additional contexts for refining the matching cost in the cost volume,

and

estimating a position and an orientation of the ADV by minimizing the matching cost in the cost volume.

20. The computer-implemented method of claim 1 , wherein the pre-built point cloud map was built using LiDAR points collected by mapping vehicles, wherein dynamic objects were removed from the pre-built point cloud map.

21. The non-transitory machine-readable medium of claim 10 , wherein the pre-built point cloud map was built using LiDAR points collected by mapping vehicles, wherein dynamic objects were removed from the pre-built point cloud map.