Systems and methods for generating synthetic light detection and ranging data via machine learning
The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.
1. A computer-implemented method to generate synthetic light detection and ranging (LiDAR) data, the method comprising:
generating, using a physics-based simulation engine and based at least in part on an object in an environment, an initial point cloud that comprises a plurality of points descriptive of the object; and
generating, using a machine-learned geometry network and based at least in part on the initial point cloud, an adjusted point cloud,
wherein the machine-learned geometry network was trained by evaluating a loss over synthetic point clouds generated using the machine-learned geometry network and ground truth point clouds collected by a physical LiDAR system, the loss configured to correspond to a perceptual similarity between the synthetic point clouds and the ground truth point clouds.
2. The computer-implemented method of claim 1 , comprising:
inputting the adjusted point cloud to a machine-learned perception system for an autonomous vehicle to simulate real-world LIDAR data; and
evaluating an output of the machine-learned perception system generated based at least in part on the adjusted point cloud.
3. The computer-implemented method of claim 1 , wherein the adjusted point cloud corresponds to a new view of the object.
4. The computer-implemented method of claim 1 , wherein the object is a virtual object inserted into the environment.
5. The computer-implemented method of claim 1 , comprising:
simulating a virtual LIDAR system moving along a trajectory through the environment,
wherein the adjusted point cloud corresponds to a simulated output of the virtual LIDAR system.
6. The computer-implemented method of claim 5 , wherein simulating the virtual LIDAR system comprises determining a ray casting location and a ray casting direction based at least in part on the trajectory, the ray casting location and the ray casting direction being used by the physics-based simulation engine to generate the initial point cloud.
7. The computer-implemented method of claim 1 , comprising:
obtaining real-world LiDAR data of the object physically collected by a LiDAR system in the environment;
converting the real-world LiDAR data to a mesh representation of the object; and
generating the initial point cloud using the mesh representation of the object.
8. A computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
generating, using a physics-based simulation engine and based at least in part on an object in an environment, an initial point cloud that comprises a plurality of points descriptive of the object; and
generating, using a machine-learned geometry network and based at least in part on the initial point cloud, an adjusted point cloud,
wherein the machine-learned geometry network was trained by evaluating a loss over synthetic point clouds generated using the machine-learned geometry network and ground truth point clouds collected by a physical LiDAR system, the loss configured to correspond to a perceptual similarity between the synthetic point clouds and the ground truth point clouds.
9. The computing system of claim 8 , wherein the operations further comprise:
inputting the adjusted point cloud to a machine-learned perception system for an autonomous vehicle to simulate real-world LIDAR data; and
evaluating an output of the machine-learned perception system generated based at least in part on the adjusted point cloud.
10. The computing system of claim 8 , wherein the adjusted point cloud corresponds to a new view of the object.
11. The computing system of claim 8 , wherein the object is a virtual object inserted into the environment.
12. The computing system of claim 11 , wherein the virtual object is a virtual vehicle inserted into the environment.
13. The computing system of claim 8 , wherein the operations further comprise:
simulating a virtual LIDAR system moving along a trajectory through the environment,
wherein the adjusted point cloud corresponds to a simulated output of the virtual LIDAR system.
14. The computing system of claim 13 , wherein simulating the virtual LIDAR system comprises determining a ray casting location and a ray casting direction based at least in part on the trajectory, the ray casting location and the ray casting direction being used by the physics-based simulation engine to generate the initial point cloud.
15. The computing system of claim 8 , wherein the operations further comprise:
obtaining real-world LiDAR data of the object physically collected by a LiDAR system in the environment;
converting the real-world LiDAR data to a mesh representation of the object; and
generating the initial point cloud using the mesh representation of the object.
16. The computing system of claim 8 , wherein the computing system is onboard an autonomous vehicle.
17. The computing system of claim 16 , wherein the autonomous vehicle is an autonomous truck.
18. A system for training geometry models for generating synthetic light detection and ranging (LiDAR) data, the system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
generating, using a physics-based simulation engine and based at least in part on an object in an environment, an initial point cloud that comprises a plurality of points descriptive of the object; and
generating, using a machine-learned geometry network and based at least in part on the initial point cloud, an adjusted point cloud;
evaluating a loss over the adjusted point cloud and a corresponding ground truth point cloud collected by a physical LiDAR system, the loss configured to correspond to a perceptual similarity between the adjusted point cloud and the corresponding ground truth point cloud collected; and
updating one or more parameters of the machine-learned geometry network based at least in part on the loss.
19. The system of claim 18 , wherein the operations further comprise:
obtaining real-world LiDAR data of the object physically collected by a LiDAR system in the environment;
converting the real-world LiDAR data to a mesh representation of the object; and
generating the initial point cloud using the mesh representation of the object.
20. The system of claim 18 , wherein the object is a virtual object inserted into the environment.