Systems and methods for generating synthetic light detection and ranging data via machine learning
The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.
1. A computer-implemented method to generate synthetic light detection and ranging (LiDAR) data, the method comprising:
obtaining, by a computing system comprising one or more computing devices, a three-dimensional map of an environment;
determining, by the computing system, a trajectory that describes a series of locations of a virtual object relative to the environment over time;
performing, by the computing system, a ray casting simulation on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points descriptive of at least a portion of the environment, wherein a respective depth is associated with a respective point of the plurality of points;
processing, by the computing system using a machine-learned geometry network, the initial three-dimensional point cloud to predict a respective adjusted depth for the respective point; and
generating, by the computing system, an adjusted three-dimensional point cloud in which the respective point has the respective adjusted depth predicted by the machine-learned geometry network.
2. The computer-implemented method of claim 1 , further comprising:
generating, by the computing system, a respective intensity value for the respective point based at least in part on intensity data included in the three-dimensional map for locations within a radius of a respective location associated with the respective point in either the initial three-dimensional point cloud or the adjusted three-dimensional point cloud.
3. The computer-implemented method of claim 1 , wherein performing, by the computing system, the ray casting to generate the initial three-dimensional point cloud comprises determining, by the computing system for a respective ray of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory.
4. The computer-implemented method of claim 3 , wherein performing, by the computing system, the ray casting to generate the initial three-dimensional point cloud comprises:
identifying, by the computing system for the respective ray, a closest surface element in the three-dimensional map to the ray casting location and along the ray casting direction; and
generating, by the computing system for the respective ray, the respective point with the respective depth based at least in part on a distance from the ray casting location to the closest surface element.
5. The computer-implemented method of claim 1 , further comprising feeding, by the computing system, the adjusted three-dimensional point cloud as LiDAR data input to an autonomy computing system of an autonomous vehicle to test a performance of the autonomy computing system of the autonomous vehicle in the environment.
6. The computer-implemented method of claim 1 , wherein the machine-learned geometry network comprises a parametric continuous convolution network.
7. The computer-implemented method of claim 1 , wherein the machine-learned geometry network comprises a plurality of continuous fusion layers with residual connections between adjacent layers.
8. The computer-implemented method of claim 1 , wherein obtaining, by the computing system, the three-dimensional map of the environment comprises generating, by the computing system, the three-dimensional map, and generating, by the computing system, the three-dimensional map comprises:
obtaining, by the computing system, a plurality of sets of real-world LiDAR data physically collected by one or more LiDAR systems in the environment;
removing, by the computing system, one or more moving objects from the plurality of sets of real-world LiDAR data;
associating, by the computing system, the plurality of sets of real-world LiDAR data to a common coordinate system to generate an aggregate LiDAR point cloud; and
converting, by the computing system, the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh.
9. The computer-implemented method of claim 1 , wherein the machine-learned geometry network has been trained using an objective function that comprises a reconstruction loss term that measures respective distances between points included in a synthetic three-dimensional point cloud generated using the machine-learned geometry network and points included in a ground truth three-dimensional point cloud collected by a physical LiDAR system.
10. The computer-implemented method of claim 1 , wherein the machine-learned geometry network has been trained using an objective function that comprises an adversarial loss term that measures an ability of a discriminator network to select which of a synthetic three-dimensional point cloud generated using the machine-learned geometry network and a ground truth three-dimensional point cloud collected by a physical LiDAR system is real and which is synthetic.
11. A computing system, comprising:
one or more processors;
a machine-learned geometry network; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a ground truth three-dimensional point cloud collected by a physical LiDAR system as the physical LiDAR system travelled along a trajectory through an environment;
obtaining a three-dimensional map of the environment;
performing a ray casting simulation on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points descriptive of at least a portion of the environment, wherein a respective depth is associated with a respective point of the plurality of points;
processing, by the machine-learned geometry network, the initial three-dimensional point cloud to predict a respective adjusted depth for the respective point;
generating an adjusted three-dimensional point cloud in which the respective point has the respective adjusted depth predicted by the machine-learned geometry network;
evaluating an objective function that compares the adjusted three-dimensional point cloud to the ground truth three-dimensional point cloud; and
modifying one or more values of one or more parameters of the machine-learned geometry network based at least in part on the objective function.
12. The computing system of claim 11 , wherein evaluating the objective function comprises determining respective distances between the plurality of points included in the adjusted three-dimensional point cloud and ground truth points included in the ground truth three-dimensional point cloud collected by the physical LiDAR system.
13. The computing system of claim 11 , wherein evaluating the objective function comprises:
providing the adjusted three-dimensional point cloud and the ground truth three-dimensional point cloud to a discriminator network;
receiving, from the discriminator network, a selection of one of the adjusted three-dimensional point cloud and the ground truth three-dimensional point cloud as real; and
determining a loss value based at least in part on the selection received from the discriminator network.
14. The computing system of claim 11 , wherein modifying the one or more values of the one or more parameters of the machine-learned geometry network based at least in part on the objective function comprises backpropagating the objective function through the machine-learned geometry network.
15. The computing system of claim 11 , wherein the machine-learned geometry network comprises a parametric continuous convolution network.
16. The computing system of claim 11 , wherein the machine-learned geometry network comprises a plurality of continuous fusion layers with residual connections between adjacent layers.
17. One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processor to perform operations, the operations comprising:
obtaining a three-dimensional map of an environment;
determining a trajectory that describes a series of locations of a virtual object relative to the environment over time;
performing a ray casting simulation on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points descriptive of at least a portion of the environment, wherein a respective depth is associated with a respective point of the plurality of points;
processing, using a machine-learned geometry network, the initial three-dimensional point cloud to predict a respective adjusted depth for the respective point; and
generating an adjusted three-dimensional point cloud in which the respective point has the respective adjusted depth predicted by the machine-learned geometry network.
18. The one or more non-transitory computer-readable media of claim 17 , wherein the operations further comprise:
generating a respective intensity value for the respective point based at least in part on intensity data included in the three-dimensional map for locations within a radius of a respective location associated with such point in either the initial three-dimensional point cloud or the adjusted three-dimensional point cloud.
19. The one or more non-transitory computer-readable media of claim 17 , wherein performing the ray casting to generate the initial three-dimensional point cloud comprises determining, for a respective ray of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory.
20. The one or more non-transitory computer-readable media of claim 17 , wherein performing the ray casting to generate the initial three-dimensional point cloud comprises:
identifying, for the respective ray, a closest surface element in the three-dimensional map to the ray casting location and along the ray casting direction; and
generating, for the respective ray, the respective point with its respective depth based at least in part on a distance from the ray casting location to the closest surface element.