Identifying objects using LiDAR
Among other things, techniques are described for controlling, using a control circuit, motion of a vehicle based objects identified using LiDAR. For example, respective classes of points of a point cloud are determined, and based on the determined respective classes of the points of the point cloud, objects in the vicinity of the vehicle are identified.
1. A method, comprising:
receiving, using at least one processor, LiDAR point cloud data associated with a plurality of points in a point cloud;
generating, using the at least one processor, a Birds-eye View (BeV) of the point cloud and a Range View (RV) of the point cloud based on the plurality of points in the point cloud, wherein the RV is different from the BeV;
providing, using the at least one processor, the BeV as input to a first view neural network and the RV as input to a RV neural network, the RV neural network different from the BeV neural network;
generating, using the at least one processor, for each point in the point cloud, a first set of class scores indicative of object classes using the BeV neural network, and a second set of class scores indicative of the object classes using the RV neural network, wherein the first set of class scores and the second set of class scores are generated in parallel;
determining, using the at least one processor, a finalized label for at least one point in the point cloud, wherein the determining is based on the first set of class scores of the at least one point and the second set of class scores of the at least one point;
identifying, using the at least one processor, at least one object in a vicinity of a vehicle based at least in part on the finalized label of the at least one point; and
controlling, using the at least one processor, motion of the vehicle based on the at least one object.
2. The method of claim 1 , wherein the determining a finalized label for at least one point in the point cloud comprises:
determining at least one uncertain point in the point cloud, wherein the determining is based on the first set of class scores of the at least one uncertain point and the second set of class scores of the at least one uncertain point;
generating, using a fusion neural network, a third set of class scores for the at least one uncertain point based on the at least one of the first set of class scores of the at least one uncertain point and the second set of class scores of the at least one uncertain point, wherein the third set of class scores is based on characteristics of neighboring points of the at least one uncertain point; and
based on the third set of class scores, determining, using the at least one processor, a finalized label of the at least one uncertain point.
3. The method of claim 1 , wherein the LiDAR point cloud data comprises information about a color associated with at least one point included in the point cloud.
4. The method of claim 1 , wherein the LiDAR point cloud data comprises point intensity information.
5. The method of claim 2 , wherein at least one class score included in the first set of class scores, the second set of class scores or the third set of class scores of a particular point corresponds to a pre-defined class of object.
6. The method of claim 2 , wherein at least one of the BeV neural network, the RV neural network, and the fusion neural network comprises at least one recurrent layer.
7. The method of claim 1 , wherein providing the BeV data as input to the BeV neural network and the RV data as input to the RV neural network comprises:
providing the BeV data as input to the BeV neural network and the RV data as input to the RV neural network, the BeV data or the RV data generated at least in part based on data augmentation.
8. The method of claim 1 , wherein the uncertain point is determined with respect to a threshold difference in class scores, wherein the threshold difference is determined based on at least one of a probability function or a filtering function or both.
9. The method of claim 1 , wherein a class score for the at least one uncertain point is determined based on concatenated features of neighboring points of the at least one uncertain point.
10. The method of claim 9 , wherein the concatenated features comprise intermediate output of intermediate layers of the BeV neural network and the RV neural network.
11. The method of claim 9 , wherein the concatenated features comprise the output class scores from the at least one of the BeV neural network and the RV neural network.
12. The method of claim 2 , wherein the fusion neural network includes at least one of a multi-layer perceptron or a convolutional layer.
13. A vehicle, comprising:
at least one LiDAR device capable of generating a LiDAR scan point cloud that includes a plurality of LiDAR data points; and
a processing circuit coupled to the LiDAR device, the processing circuit configured for:
receiving LiDAR point cloud data associated with a plurality of points in a point cloud;
generating a Birds-eye View (BeV) of the point cloud and a Range View (RV) of the point cloud based on the plurality of points in the point cloud, wherein the RV is different from the BeV;
providing the BeV as input to a BeV neural network and the RV as input to a RV neural network, the RV neural network different from the BeV neural network;
generating for each point in the point cloud, a first set of class scores indicative of object classes using the BeV neural network, and a second set of class scores indicative of the object classes using the RV neural network, wherein the first set of class scores and the second set of class scores are generated in parallel;
determining a finalized label for at least one point in the point cloud, wherein the determining is based on the first set of class scores of the at least one point and the second set of class scores of the at least one point;
identifying at least one object in a vicinity of a vehicle based at least in part on the finalized label of the at least one point; and
controlling motion of the vehicle based on the at least one object.
14. The vehicle of claim 13 , wherein determining a finalized label for at least one point in the point cloud comprises:
determining at least one uncertain point in the point cloud, wherein the determining is based on the first set of class scores of the at least one uncertain point and the second set of class scores of the at least one uncertain point;
generating, using a fusion neural network, a third set of class scores for the at least one uncertain point based on the at least one of the first set of class scores of the at least one uncertain point and the second set of class scores of the at least one uncertain point, wherein the third set of class scores is based on characteristics of neighboring points of the at least one uncertain point; and
based on the third set of class scores, determining a finalized label of the at least one uncertain point.
15. The vehicle of claim 13 , wherein the LiDAR point cloud data comprises information about a color associated with at least one point included in the point cloud.
16. The vehicle of claim 13 , wherein the LiDAR point cloud data comprises point intensity information.
17. The vehicle of claim 14 , wherein at least one class score included in the first set of class scores, the second set of class scores or the third set of class scores of a particular point corresponds to a pre-defined class of object.
18. The vehicle of claim 14 , wherein at least one of the BeV neural network, the RV neural network, and the fusion neural network comprises at least one recurrent layer.
19. The vehicle of claim 13 , wherein providing the BeV data as input to the BeV neural network and the RV data as input to the RV neural network comprises:
providing the BeV data as input to the BeV neural network and the RV data as input to the RV neural network, the BeV data or the RV data generated at least in part based on data augmentation.
20. The vehicle of claim 13 , wherein the uncertain point is determined with respect to a threshold difference in class scores, wherein the threshold difference is determined based on at least one of a probability function or a filtering function or both.
21. The vehicle of claim 13 , wherein a class score for the at least one uncertain point is determined based on concatenated features of neighboring points of the at least one uncertain point.
22. The vehicle of claim 21 , wherein the concatenated features comprise intermediate output of intermediate layers of the BeV neural network and the RV neural network.
23. The vehicle of claim 21 , wherein the concatenated features comprise the output class scores from the BeV neural network and the RV neural network.
24. The vehicle of claim 14 , wherein the fusion neural network includes at least one of a multi-layer perceptron or a convolutional layer.