Systems and methods for generating synthetic sensor data via machine learning
The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.
1. A computer-implemented method comprising:
obtaining a three-dimensional map of an environment;
determining a trajectory that describes a series of locations of a virtual object relative to the environment over time;
performing a ray casting simulation on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points descriptive of at least a portion of the environment, wherein performing the ray casting simulation on the three-dimensional map according to the trajectory comprises determining, for each of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory;
processing, using a machine-learned model, the initial three-dimensional point cloud to predict a respective probability of an error for a respective point of the plurality of points of the initial three-dimensional point cloud; and
generating an adjusted three-dimensional point cloud from the initial three-dimensional point cloud by modifying geometry of the initial three-dimensional point cloud to include the error based at least in part on the respective probability of the error.
2. The computer-implemented method of claim 1 , wherein the respective probability of the error comprises a dropout probability.
3. The computer-implemented method of claim 2 , wherein generating the adjusted three-dimensional point cloud from the initial three-dimensional point cloud comprises removing the respective point based on the respective probability of the error for the respective point.
4. The computer-implemented method of claim 2 , wherein processing, using the machine-learned model, the initial three-dimensional point cloud to predict the respective probability of the error for the respective point of the plurality of points of the initial three-dimensional point cloud comprises:
transforming the initial three-dimensional point cloud into a two-dimensional polar image grid; and
processing, using the machine-learned model, the two-dimensional polar image grid to generate a two-dimensional ray dropout probability map.
5. The computer-implemented method of claim 1 , wherein performing the ray casting simulation on the three-dimensional map according to the trajectory to generate the initial three-dimensional point cloud comprises:
identifying, for each of the plurality of rays, a closest surface element in the three-dimensional map to the ray casting location and along the ray casting direction; and
generating, for each of the plurality of rays, one of the plurality of points with a respective depth based at least in part on a distance from the ray casting location to the closest surface element.
6. The computer-implemented method of claim 1 , further comprising feeding the adjusted three-dimensional point cloud as LiDAR data input to an autonomy computing system of an autonomous vehicle to test a performance of the autonomy computing system of the autonomous vehicle in the environment.
7. The computer-implemented method of claim 1 , wherein obtaining the three-dimensional map of the environment comprises generating the three-dimensional map, and wherein generating the three-dimensional map comprises:
obtaining a plurality of sets of real-world LiDAR data physically collected by one or more LiDAR systems in the environment;
removing one or more moving objects from the plurality of sets of real-world LiDAR data;
associating the plurality of sets of real-world LiDAR data to a common coordinate system to generate an aggregate LiDAR point cloud; and
converting the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh.
8. The computer-implemented method of claim 1 , wherein the machine-learned model has been trained using an objective function that comprises a pixel-wise loss that compares a predicted dropout probability map with a ground truth dropout mask.
9. The computer-implemented method of claim 1 , further comprising:
inserting one or more dynamic virtual objects into the three-dimensional map of the environment;
wherein performing ray casting on the three-dimensional map comprises performing ray casting on the three-dimensional map including the one or more dynamic virtual objects.
10. A computing system, comprising:
one or more processors;
a machine-learned model configured to predict probabilities of error for LiDAR data; and
one or more non-transitory computer-readable media that store instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining a ground truth three-dimensional point cloud collected by a physical LiDAR system as the physical LiDAR system travelled along a trajectory through an environment;
generating a ground truth mask for the ground truth three-dimensional point cloud;
obtaining a three-dimensional map of the environment;
performing ray casting on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points;
processing, using the machine-learned model, the initial three-dimensional point cloud to generate a probability of error map that provides a respective probability of error for one or more of the plurality of points of the initial three-dimensional point cloud;
evaluating an objective function that compares the probability of error map generated by the machine-learned model to the ground truth mask; and
modifying one or more values of one or more parameters of the machine-learned model based at least in part on the objective function.
11. The computing system of claim 10 , wherein the ground truth mask comprises a ground truth dropout mask and the probability of error map comprises a dropout probability map.
12. The computing system of claim 11 , wherein each of the ground truth dropout mask and the dropout probability map comprises a two-dimensional polar image grid.
13. The computing system of claim 11 , wherein evaluating the objective function comprises determining a pixel-wise binary cross entropy between the ground truth dropout mask and the dropout probability map.
14. The computing system of claim 10 , wherein modifying the one or more values of the one or more parameters of the machine-learned model based at least in part on the objective function comprises backpropagating the objective function through the machine-learned model.
15. One or more non-transitory computer-readable media that store instructions that are executable by a computing system comprising one or more computing devices to cause the computing system to generate three-dimensional representations of objects by performing operations, the operations comprising:
obtaining one or more sets of real-world LiDAR data physically collected by one or more LiDAR systems in a real-world environment, the one or more sets of real-world LiDAR data respectively comprising one or more three-dimensional point clouds;
defining a three-dimensional bounding box for an object included in the real-world environment;
identifying points from the one or more three-dimensional point clouds that are included within the three-dimensional bounding box to generate a set of accumulated points; and
generating a three-dimensional model of the object based at least in part on the set of accumulated points by:
mirroring the set of accumulated points along at least one axis of the three-dimensional bounding box to generate a set of mirrored points;
concatenating the set of mirrored points with the set of accumulated points to generate a set of object points associated with the object; and
generating the three-dimensional model based at least in part on the set of object points.
16. The one or more non-transitory computer-readable media of claim 15 , wherein generating the three-dimensional model comprises:
generating a mesh representation of the object from the set of object points associated with the object.
17. The one or more non-transitory computer-readable media of claim 16 , wherein generating the mesh representation of the object comprises performing surfel-disk reconstruction on the set of object points.
18. The one or more non-transitory computer-readable media of claim 15 , wherein the operations comprise:
associating intensity data obtained from the one or more sets of real-world LiDAR data with the three-dimensional model.
19. The one or more non-transitory computer-readable media of claim 15 , wherein the operations comprise:
performing a ray casting simulation on the three-dimensional model to generate a three-dimensional point cloud descriptive of the object; and
feeding the three-dimensional point cloud as LiDAR data input to an autonomy computing system for an autonomous vehicle to test a performance of the autonomy computing system of the autonomous vehicle.
20. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining a three-dimensional map of an environment;
determining a trajectory that describes a series of locations of a virtual object relative to the environment over time;
performing a ray casting simulation on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points descriptive of at least a portion of the environment, wherein performing the ray casting simulation on the three-dimensional map according to the trajectory comprises determining, for each of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory;
processing, using a machine-learned model, the initial three-dimensional point cloud to predict a respective probability of an error for a respective point of the plurality of points of the initial three-dimensional point cloud; and
generating an adjusted three-dimensional point cloud from the initial three-dimensional point cloud by modifying geometry of the initial three-dimensional point cloud to include the error based at least in part on the respective probability of the error.
21. One or more non-transitory computer-readable media that store instructions that are executable by one or more processors to cause a computing system to perform operations, the operations comprising:
obtaining a three-dimensional map of an environment;
determining a trajectory that describes a series of locations of a virtual object relative to the environment over time;
performing a ray casting simulation on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points descriptive of at least a portion of the environment, wherein performing the ray casting simulation on the three-dimensional map according to the trajectory comprises determining, for each of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory;
processing, using a machine-learned model, the initial three-dimensional point cloud to predict a respective probability of an error for a respective point of the plurality of points of the initial three-dimensional point cloud; and
generating an adjusted three-dimensional point cloud from the initial three-dimensional point cloud by modifying geometry of the initial three-dimensional point cloud to include the error based at least in part on the respective probability of the error.
22. One or more non-transitory computer-readable media that store instructions that are executable by one or more processors to cause a computing system to perform operations, the operations comprising:
obtaining a ground truth three-dimensional point cloud collected by a physical LiDAR system as the physical LiDAR system travelled along a trajectory through an environment;
generating a ground truth mask for the ground truth three-dimensional point cloud;
obtaining a three-dimensional map of the environment;
performing ray casting on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points;
processing, using a machine-learned model configured to predict probabilities of error for LiDAR data, the initial three-dimensional point cloud to generate a probability of error map that provides a respective probability of error for one or more of the plurality of points of the initial three-dimensional point cloud;
evaluating an objective function that compares the probability of error map generated by the machine-learned model to the ground truth mask; and
modifying one or more values of one or more parameters of the machine-learned model based at least in part on the objective function.
23. A computer-implemented method, comprising:
obtaining a ground truth three-dimensional point cloud collected by a physical LiDAR system as the physical LiDAR system travelled along a trajectory through an environment;
generating a ground truth mask for the ground truth three-dimensional point cloud;
obtaining a three-dimensional map of the environment;
performing ray casting on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points;
processing, using a machine-learned model configured to predict probabilities of error for LiDAR data, the initial three-dimensional point cloud to generate a probability of error map that provides a respective probability of error for one or more of the plurality of points of the initial three-dimensional point cloud;
evaluating an objective function that compares the probability of error map generated by the machine-learned model to the ground truth mask; and
modifying one or more values of one or more parameters of the machine-learned model based at least in part on the objective function.
24. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining one or more sets of real-world LiDAR data physically collected by one or more LiDAR systems in a real-world environment, the one or more sets of real-world LiDAR data respectively comprising one or more three-dimensional point clouds;
defining a three-dimensional bounding box for an object included in the real-world environment;
identifying points from the one or more three-dimensional point clouds that are included within the three-dimensional bounding box to generate a set of accumulated points; and
generating a three-dimensional model of the object based at least in part on the set of accumulated points by:
mirroring the set of accumulated points along at least one axis of the three-dimensional bounding box to generate a set of mirrored points;
concatenating the set of mirrored points with the set of accumulated points to generate a set of object points associated with the object; and
generating the three-dimensional model based at least in part on the set of object points.
25. A computer-implemented method, comprising:
obtaining one or more sets of real-world LiDAR data physically collected by one or more LiDAR systems in a real-world environment, the one or more sets of real-world LiDAR data respectively comprising one or more three-dimensional point clouds;
defining a three-dimensional bounding box for an object included in the real-world environment;
identifying points from the one or more three-dimensional point clouds that are included within the three-dimensional bounding box to generate a set of accumulated points; and
generating a three-dimensional model of the object based at least in part on the set of accumulated points by:
mirroring the set of accumulated points along at least one axis of the three-dimensional bounding box to generate a set of mirrored points;
concatenating the set of mirrored points with the set of accumulated points to generate a set of object points associated with the object; and
generating the three-dimensional model based at least in part on the set of object points.