Vehicle semantic keypoint point cloud definition

1 . A method of identifying keypoints associated with an object based on Light Detection and Ranging (LiDAR) point cloud data, the method comprising:

inputting the LiDAR point cloud data representing an object as perceived by a LiDAR sensor of a vehicle into an algorithm trained to identify the keypoints associated with the object wherein the keypoints represent localizable and trackable physical aspects of the object;

identifying, by the algorithm, at least one keypoint associated with the object and a respective confidence score for the at least one keypoint;

determining, based on the at least one keypoint, a navigation path for the vehicle;

navigating the vehicle along the navigation path; and

training a machine learning algorithm to result in the algorithm trained to identify the keypoints associated with the object, wherein training the machine learning algorithm comprises:

creating a labeled training dataset by semantically labeling parts of the object represented in LiDAR point clouds as keypoints;

inserting a LiDAR point cloud from the labeled training dataset into the machine learning algorithm;

receiving an output from the machine learning algorithm identifying keypoints and semantic labels associated with respective keypoints;

providing a loss value to the machine learning algorithm indicating whether the keypoints were properly located as compared to the labeled training dataset; and

providing a loss value to the machine learning algorithm indicating whether the semantic labels were properly identified as compared to the labeled training dataset.

2 . The method of claim 1 , wherein training the machine learning algorithm further comprises:

processing an existing dataset by inserting training keypoints to generate a labeled training dataset, wherein the existing dataset includes a plurality of LiDAR point clouds and ground-truth-labeled bounding boxes around objects in the LiDAR point clouds, and the training keypoints are inserted on locations on the ground-truth-labeled bounding boxes, wherein the training keypoints represent localizable and trackable physical aspects of the locations on the ground-truth-labeled bounding boxes;

inserting a LiDAR point cloud from the existing dataset into the machine learning algorithm;

receiving an output from the machine learning algorithm including a bounding box around the object in the LiDAR point cloud, keypoints aligned to the bounding box, and respective confidence scores associated with the keypoints aligned to the bounding box; and

generating a loss value to the machine learning algorithm indicating whether the keypoints were properly located as compared to the labeled training dataset.

3 . The method of claim 2 , wherein the locations on the ground-truth-labeled bounding boxes are corners of the ground-truth-labeled bounding boxes.

4 . The method of claim 2 , wherein processing the existing dataset by inserting the training keypoints to result in the labeled training dataset further comprises:

providing a respective training confidence score associated with each of the training keypoints.

5 . The method of claim 4 , wherein determining each of the respective confidence scores associated with the keypoints comprises:

determining a distance from one of the keypoints aligned to the bounding box to a nearest LiDAR point in the LiDAR point cloud data that is associated with the object; and

representing the confidence score as a value indicative of the distance from the one of the keypoints to the nearest LiDAR point associated with the object, wherein a greater distance is correlated to a lower confidence score.

6 . The method of claim 5 , wherein the training the machine learning algorithm further comprises:

generating a loss value to the machine learning algorithm indicating whether the confidence score associated with its respective keypoint is correct.

7 . The method of claim 6 , wherein a result of associating the confidence score to the distance of the keypoint on the object is that the confidence score is a proxy for how visible the object is near the keypoint, and thereby how confident the trained machine learning algorithm can be of the location of that keypoint, wherein this has a consequence that the representation of the object in keypoints emphasizes the location of observable features over unobservable features.

8 . The method of claim 2 , wherein the locations to which the keypoints are inserted are mid-points of lines making up the ground-truth-labeled bounding boxes.

9 . The method of claim 1 , wherein training the machine learning algorithm further comprises:

inserting a LiDAR point cloud into a self-supervised machine learning algorithm, wherein the self-supervised machine learning algorithm is configured to identify keypoints and semantic labels associated with respective keypoints,

optimizing keypoint identification such that keypoints that are located on similar objects at similar locations reduce a variance value for locations of detected keypoints on similar objects, and

optimizing keypoint identification such that keypoints, when tracked over a series of frames, identify object kinematics that closely correspond to a function representing smoothed object kinematics over the series of frames,

whereby the machine learning algorithm trained to identify the keypoints associated with the object is optimized to identify keypoints that are consistently identifiable and result in a better observed object kinematics,

wherein the keypoints do not have a semantic value associated with them.

10 . A system comprising:

a processor; and

a non-transitory computer readable medium storing instructions that, when executed by the processor, cause the processor to:

input LiDAR point cloud data representing an object as perceived by a LiDAR sensor of a vehicle into an algorithm trained to identify keypoints associated with the object, wherein the keypoints represent localizable and trackable physical aspects of the object;

identify, by the algorithm, at least one keypoint associated with the object and a respective confidence score for the at least one keypoint;

determine, based on the at least one keypoint, a navigation path for the vehicle; and

navigate the vehicle along the navigation path,

wherein the algorithm comprises a machine learning algorithm, and training the machine learning algorithm comprises:

creating a labeled training dataset by semantically labeling parts of the object represented in LiDAR point clouds as keypoints;

inserting a LiDAR point cloud from the labeled training dataset into the machine learning algorithm;

receiving an output from the machine learning algorithm identifying keypoints and semantic labels associated with respective keypoints;

providing a loss value to the machine learning algorithm indicating whether the keypoints were properly located as compared to the labeled training dataset; and

providing a loss value to the machine learning algorithm indicating whether the semantic labels were properly identified as compared to the labeled training dataset.

11 . The system of claim 10 , wherein training the machine learning algorithm further comprises:

processing an existing dataset by inserting training keypoints to generate a labeled training dataset, wherein the existing dataset includes a plurality of LiDAR point clouds and ground-truth-labeled bounding boxes around objects in the LiDAR point clouds, and the training keypoints are inserted on locations on the ground-truth-labeled bounding boxes wherein the training keypoints represent localizable and trackable physical aspects of the locations on ground-truth-labeled bounding-boxes;

inserting a LiDAR point cloud from the existing dataset into the machine learning algorithm;

receiving an output from the machine learning algorithm including a bounding box around the object in the LiDAR point cloud, keypoints aligned to the bounding box, and respective confidence scores associated with the keypoints aligned to the bounding box; and

generating a loss value to the machine learning algorithm indicating whether the keypoints were properly located as compared to the labeled training dataset.

12 . The system of claim 11 , wherein the locations to which the keypoints are inserted are mid-points of lines making up the ground-truth-labeled bounding boxes.

13 . The system of claim 10 , wherein training the machine learning algorithm further comprises:

inserting a LiDAR point cloud into a self-supervised machine learning algorithm, wherein the self-supervised machine learning algorithm is configured to identify keypoints and semantic labels associated with respective keypoints;

optimizing keypoint identification such that keypoints that are located on similar objects at similar locations reduce a variance value for locations of detected keypoints on similar objects; and

optimizing keypoint identification such that keypoints, when tracked over a series of frames, identify object kinematics that closely correspond to a function representing smoothed object kinematics over the series of frames,

wherein the machine learning algorithm trained to identify the keypoints associated with the object is optimized to identify keypoints that are consistently identifiable and result in a better observed object kinematics, and

wherein the keypoints do not have a semantic value associated with them.

14 . A non-transitory computer readable medium comprising instructions, the instructions, when executed by a computing system, cause the computing system to:

input LiDAR point cloud data representing an object as perceived by a LiDAR sensor of a vehicle into an algorithm trained to identify keypoints associated with the object, wherein the keypoints represent localizable and trackable physical aspects of the object;

identify, by the algorithm, at least one keypoint associated with the object and a respective confidence score for the at least one keypoint;

determine, based on the at least one keypoint, a navigation path for the vehicle; and

navigate the vehicle along the navigation path,

wherein the algorithm comprises a machine learning algorithm, and training the machine learning algorithm comprises:

inserting a LiDAR point cloud into a self-supervised machine learning algorithm, wherein the self-supervised machine learning algorithm is configured to identify keypoints and semantic labels associated with respective keypoints;

optimizing keypoint identification such that keypoints that are located on similar objects at similar locations reduce a variance value for locations of detected keypoints on similar objects; and

optimizing keypoint identification such that keypoints, when tracked over a series of frames, identify object kinematics that closely correspond to a function representing smoothed object kinematics over the series of frames,

wherein the machine learning algorithm trained to identify the keypoints associated with the object is optimized to identify keypoints that are consistently identifiable and result in a better observed object kinematics, and

wherein the keypoints do not have a semantic value associated with them.

15 . The computer readable medium of claim 14 , wherein training the machine learning algorithm further comprises:

processing an existing dataset by insert training keypoints to generate a labeled training dataset, wherein the existing dataset includes a plurality of LiDAR point clouds and ground-truth-labeled bounding boxes around objects in the LiDAR point clouds, and the training keypoints are inserted on locations on the ground-truth-labeled bounding boxes, wherein the training keypoints represent localizable and trackable physical aspects of the locations on the ground-truth-labeled bounding boxes;

inserting a LiDAR point cloud from the exist dataset into the machine learning algorithm;

receiving an output from the machine learning algorithm including a bounding box around the object in the LiDAR point cloud, keypoints aligned to the bounding box, and respective confidence scores associated with the keypoints aligned to the bounding box; and

generating a loss value to the machine learning algorithm indicating whether the keypoints were properly located as compared to the labeled training dataset.

16 . The computer readable medium of claim 15 , wherein the locations to which the keypoints are inserted are mid-points of lines making up the ground-truth-labeled bounding boxes.

17 . The computer readable medium of claim 14 , wherein training the machine learning algorithm further comprises:

creating a labeled training dataset by semantically labeling parts of the object represented in LiDAR point clouds as keypoints;

inserting a LiDAR point cloud from the labeled training dataset into the machine learning algorithm;

receiving an output from the machine learning algorithm identifying keypoints and semantic labels associated with respective keypoints;

providing a loss value to the machine learning algorithm indicating whether the keypoints were properly located as compared to the labeled training dataset; and

providing a loss value to the machine learning algorithm indicating whether the semantic labels were properly identified as compared to the labeled training dataset.