Systems and methods for generating synthetic light detection and ranging data via machine learning
The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.
1. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing:
one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated using a physics-based simulation engine to obtain an initial point cloud and using a machine-learned simulation model to adjust the initial point cloud to cause the synthetic LiDAR data to appear more realistic; and
instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, the operations comprising:
obtaining LiDAR data descriptive of an environment of the autonomous vehicle;
determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and
controlling the autonomous vehicle according to the motion plan.
2. The autonomous vehicle control system of claim 1 , wherein the at least one of the machine-learned models comprises a machine-learned perception model.
3. The autonomous vehicle control system of claim 2 , wherein the machine-learned perception model was tested by inputting the synthetic LiDAR data to the machine-learned perception model and evaluating an output of the machine-learned perception model.
4. The autonomous vehicle control system of claim 1 , wherein the machine-learned simulation model was trained by evaluating a loss over synthetic point clouds generated using the machine-learned simulation model and ground truth point clouds collected by a physical LiDAR system.
5. The autonomous vehicle control system of claim 4 , wherein the loss is configured to correspond to a perceptual similarity between the synthetic point clouds and the ground truth point clouds.
6. The autonomous vehicle control system of claim 1 , wherein the machine-learned simulation model comprises a machine-learned geometry network configured to adjust a geometry of the initial point cloud.
7. The autonomous vehicle control system of claim 1 , wherein the synthetic LiDAR data comprises a plurality of points descriptive of an object in an environment, wherein the object is a virtual object inserted into the environment.
8. The autonomous vehicle control system of claim 1 , wherein the machine-learned simulation model is configured to correct artifacts in the initial point cloud.
9. One or more non-transitory computer-readable media storing:
one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated using a physics-based simulation engine to obtain an initial point cloud and using a machine-learned simulation model to adjust the initial point cloud to cause the synthetic LiDAR data to appear more realistic; and
instructions that are executable by one or more processors to cause an autonomous vehicle control system to perform operations, the operations comprising:
obtaining LiDAR data descriptive of an environment of the autonomous vehicle;
determining, using the one or more machine-learned models and based on the LIDAR data, a motion plan for controlling the autonomous vehicle; and
controlling the autonomous vehicle according to the motion plan.
10. The one or more non-transitory computer-readable media of claim 9 , wherein the at least one of the machine-learned models comprises a machine-learned perception model.
11. The one or more non-transitory computer-readable media of claim 10 , wherein the machine-learned perception model was tested by inputting the synthetic LiDAR data to the machine-learned perception model and evaluating an output of the machine-learned perception model.
12. The one or more non-transitory computer-readable media of claim 9 , wherein the machine-learned simulation model was trained by evaluating a loss over synthetic point clouds generated using the machine-learned simulation model and ground truth point clouds collected by a physical LiDAR system.
13. The one or more non-transitory computer-readable media of claim 12 , wherein the loss is configured to correspond to a perceptual similarity between the synthetic point clouds and the ground truth point clouds.
14. The one or more non-transitory computer-readable media of claim 9 , wherein the machine-learned simulation model comprises a machine-learned geometry network configured to adjust a geometry of the initial point cloud.
15. The one or more non-transitory computer-readable media of claim 9 , wherein the machine-learned simulation model is configured to correct artifacts in the initial point cloud.
16. A method, comprising:
obtaining light detection and ranging (LiDAR) data descriptive of an environment of an autonomous vehicle;
determining, using one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle, wherein at least one of the one or more machine-learned models was tested using synthetic LiDAR data, wherein the synthetic LiDAR data was generated using a physics-based simulation engine to obtain an initial point cloud and using a machine-learned simulation model to adjust the initial point cloud to cause the synthetic LiDAR data to appear more realistic; and
controlling the autonomous vehicle according to the motion plan.
17. The method of claim 16 , wherein the at least one of the machine-learned models comprises a machine-learned perception model.
18. The method of claim 17 , wherein the machine-learned perception model was tested by inputting the synthetic LiDAR data to the machine-learned perception model and evaluating an output of the machine-learned perception model.
19. The method of claim 16 , wherein the machine-learned simulation model was trained by evaluating a loss over synthetic point clouds generated using the machine-learned simulation model and ground truth point clouds collected by a physical LiDAR system.
20. The method of claim 16 , wherein the machine-learned simulation model comprises a machine-learned geometry network configured to adjust a geometry of the initial point cloud.
21. The method of claim 16 , wherein the machine-learned simulation model is configured to correct artifacts in the initial point cloud.