Bird's eye view based velocity estimation via self-supervised learning
Systems and methods determining velocity of an object associated with a three-dimensional (3D) scene may include: a LIDAR system generating two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps; a pillar feature network encoding data of the point cloud data to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets in the form of pseudo images, wherein the 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and the 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set; and a feature pyramid network encoding the pillar features and performing a 2D optical flow estimation to estimate the velocity of the object.
1. A method for determining velocity of an object associated with a three-dimensional (3D) scene, the method comprising:
receiving two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps;
encoding data of the point cloud data sets using a pillar feature network to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data, wherein first 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and second 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set;
aggregating the first 2D bird's-eye-view embeddings and the second 2D bird's-eye-view embeddings into an aggregated data set;
performing a 2D optical flow estimation using an optical flow network to estimate the velocity of the object; and
using the flow estimation to perform self-supervised learning.
2. The method of claim 1 , wherein aggregating comprises grouping similar pillar features and representing a set of grouped pillar features as a single feature.
3. The method of claim 2 , further comprising performing the self-supervised learning on the aggregated data set and applying predictions from the self-supervised learning to full sets of data from the first 2D bird's-eye-view embeddings.
4. The method of claim 1 , further comprising applying a mask to the aggregated data set to mask redundant pillar features.
5. The method of claim 4 , wherein the mask comprises at least one of a dynamic mask and a static mask.
6. The method of claim 1 , wherein using the flow estimation to perform self-supervised learning comprises minimizing a distance between the first and second bird's-eye-view embeddings and learning to predict a bird's-eye-view flow estimator that is consistent with motion of the pillar features without needing data labels.
7. The method of claim 1 , wherein the 2D optical flow estimation comprises performing a forward flow estimate for flow from the first 2D bird's-eye-view embeddings to the second 2D bird's-eye-view embeddings and a reverse flow estimate for flow from the second 2D bird's-eye-view embeddings to the first 2D bird's-eye-view embeddings.
8. The method of claim 1 , wherein self-supervised learning comprises minimizing a cost function for each of a plurality of hierarchical resolutions for a forward and backward flow for a feature pyramid.
9. The method of claim 1 , wherein receiving two sets of 3D point cloud data of the scene comprises receiving the first point cloud data set by a first pillar feature network and receiving a second point cloud data set by a second pillar feature network, wherein the first point cloud data set represents the scene at a time t-1 and the second point cloud data set represents the scene at a time t subsequent to the time t-1.
10. The method of claim 1 , wherein encoding data of the point cloud data sets comprises voxelizing the point cloud data sets to render surfaces in the data sets onto a grid of discretized volume elements in a 3D space to create a set of pillars.
11. The method of claim 1 , further comprising warping a pseudo image of the first point cloud data set to align the pseudo image of the first point cloud data set with a pseudo image of the second point cloud data set.
12. The method of claim 11 , further comprising warping the pseudo image of the second point cloud data set to align the pseudo image of the second point cloud data set with the pseudo image of the first point cloud data set.
13. The method of claim 11 , wherein the 2D optical flow estimation further comprises computing a cost function of the warped pseudo image of the first point cloud data set and the pseudo image of the second point cloud data set, by identifying displacement of a feature from the first image to the second image.
14. The method of claim 13 , wherein performing a 2D optical flow estimation to estimate the velocity of the object comprises aggregating bird's eye view motion vectors to compute a single mean velocity and co-variance for the feature.
15. The method of claim 1 , wherein the method is performed using three or more sets of 3D point cloud data of the scene, and wherein encoding data of the point cloud data sets further comprises:
aligning all of the point cloud data sets into the same coordinate frame;
encoding data of each of the point cloud data sets using a pillar feature network to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets comprising pillar features for each point cloud data set; and
encoding the pillar features using a feature pyramid network and performing a 2D optical flow estimation to estimate the velocity of the object.
16. The method of claim 1 , further comprising filtering the point cloud datasets using a ground height map, wherein the filtering comprises comparing data point heights against ground height and discarding a data point whose point height is not greater than the ground height at the data point's location.
17. A system for determining velocity of an object associated with a three-dimensional (3D) scene, the system comprising:
a non-transitory memory configured to store instructions; and
at least one processor configured to execute the instructions to perform the operations of:
receiving two sets of 3D point cloud data of the scene from two consecutive point cloud sweeps;
encoding data of the point cloud data sets using a pillar feature network to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data, wherein first 2D bird's-eye-view embeddings for a first of the two point cloud data sets comprises pillar features for the first point cloud data set and second 2D bird's-eye-view embeddings for a second of the two point cloud data sets comprises pillar features for the second point cloud data set;
aggregating the first 2D bird's-eye-view embeddings and the second 2D bird's-eye-view embeddings into an aggregated data set;
performing a 2D optical flow estimation using an optical flow network to estimate the velocity of the object; and
using the flow estimation to perform self-supervised learning.
18. The system of claim 17 , wherein aggregating comprises grouping similar pillar features and representing a set of grouped pillar features as a single feature.
19. The system of claim 18 , wherein the operations further comprise performing the self-supervised learning on the aggregated data set and applying predictions from the self-supervised learning to full sets of data from the first 2D bird's-eye-view embeddings.
20. The system of claim 17 , wherein the operations further comprise applying a mask to the aggregated data set to mask redundant pillar features.
21. The system of claim 20 , wherein the mask comprises at least one of a dynamic mask and a static mask.
22. The system of claim 17 , wherein using the flow estimation to perform self-supervised learning comprises minimizing a distance between the first and second bird's-eye-view embeddings and learning to predict a bird's-eye-view flow estimator that is consistent with motion of the pillar features without needing data labels.
23. The system of claim 17 , wherein the 2D optical flow estimation comprises performing a forward flow estimate for flow from the first 2D bird's-eye-view embeddings to the second 2D bird's-eye-view embeddings and a reverse flow estimate for flow from the second 2D bird's-eye-view embeddings to the first 2D bird's-eye-view embeddings.
24. The system of claim 17 , wherein self-supervised learning comprises minimizing a cost function for each of a plurality of hierarchical resolutions for a forward and backward flow for a feature pyramid.
25. The system of claim 17 , wherein receiving two sets of 3D point cloud data of the scene comprises receiving the first point cloud data set by a first pillar feature network and receiving a second point cloud data set by a second pillar feature network, wherein the first point cloud data set represents the scene at a time t-1 and the second point cloud data set represents the scene at a time t subsequent to the time t-1.
26. The system of claim 17 , wherein encoding data of the point cloud data sets comprises voxelizing the point cloud data sets to render surfaces in the data sets onto a grid of discretized volume elements in a 3D space to create a set of pillars.
27. The system of claim 17 , wherein the operations further comprise warping a pseudo image of the first point cloud data set to align the pseudo image of the first point cloud data set with the pseudo image of the second point cloud data set.
28. The system of claim 27 , wherein the operations further comprise warping the pseudo image of the second point cloud data set to align the pseudo image of the second point cloud data set with the pseudo image of the first point cloud data set.
29. The system of claim 27 , wherein the 2D optical flow estimation further comprises computing a cost function of the warped pseudo image of the first point cloud data set and the pseudo image of the second point cloud data set, by identifying displacement of a feature from the first image to the second image.
30. The system of claim 29 , wherein performing a 2D optical flow estimation to estimate the velocity of the object comprises aggregating bird's eye view motion vectors to compute a single mean velocity and co-variance for the feature.
31. The system of claim 17 , wherein the operations are performed using three or more sets of 3D point cloud data of the scene, and wherein encoding data of the point cloud data sets further comprises:
including aligning all of the point cloud data sets into the same coordinate frame;
encoding data of each of the point cloud data sets using a pillar feature network to extract two-dimensional (2D) bird's-eye-view embeddings for each of the point cloud data sets comprising pillar features for each point cloud data set; and
encoding the pillar features using a feature pyramid network and performing a 2D optical flow estimation to estimate the velocity of the object.
32. The system of claim 17 , wherein filtering the sets of 3D point cloud data comprises comparing data point heights against ground height and discarding a data point whose point height is not greater than the ground height at the data point's location.