Photorealistic image simulation with geometry-aware composition
The present disclosure provides systems and methods for generating photorealistic image simulation data with geometry-aware composition for testing autonomous vehicles. In particular, aspects of the present disclosure can involve the intake of data on an environment and output of augmented data on the environment with the photorealistic addition of an object. As one example, data on the driving experiences of a self-driving vehicle can be augmented to add another vehicle into the collected environment data. The augmented data may then be used to test safety features of software for a self-driving vehicle.
1. One or more non-transitory computer-readable media that store instructions that are executable by a computing system to cause the computing system to perform operations for training object reconstruction, the operations comprising:
obtaining first object data, wherein the first object data comprises image data, and second object data, wherein the second object data comprises Light Detection and Ranging (LiDAR) point cloud data;
processing the first object data with a first embedding model to generate a first feature embedding;
processing the second object data with a second embedding model to generate a second feature embedding;
processing the first feature embedding and the second feature embedding with a third embedding model to generate a three-dimensional mesh model of an object based at least in part on the first feature embedding and the second feature embedding;
generating predicted image data based at least in part on the three-dimensional mesh model, wherein the predicted image data comprises a predicted reconstruction of the first object data;
generating predicted point cloud data based at least in part on the three-dimensional mesh model, wherein the predicted point cloud data comprises a predicted reconstruction of the second object data;
evaluating a loss function that compares the predicted image data and the predicted point cloud data with the first object data and the second object data, wherein the loss function comprises a silhouette loss and a Light Detection and Ranging (LiDAR) loss, wherein the silhouette loss compares a generated predicted silhouette based at least in part on the predicted image data against a generated ground truth silhouette based at least in part on the first object data, wherein the LiDAR loss compares vertices of the predicted point cloud data against ground truth vertices data based at least in part on the second object data; and
modifying parameters of at least one of the first embedding model and the second embedding model based on the loss function.
2. The one or more non-transitory computer-readable media of claim 1 , wherein the first object data comprises data from a plurality of view angles.
3. The one or more non-transitory computer-readable media of claim 1 , wherein the first object data and the second object data further comprise one or more three-dimensional bounding boxes.
4. The one or more non-transitory computer-readable media of claim 1 , wherein the first embedding model comprises a fully convolutional neural network.
5. The one or more non-transitory computer-readable media of claim 1 , wherein the second embedding model comprises a fully convolutional neural network.
6. The one or more non-transitory computer-readable media of claim 1 , wherein processing the first feature embedding and the second feature embedding with a third embedding model to generate the three-dimensional mesh model of the object based at least in part on the first feature embedding and the second feature embedding, comprises processing the first feature embedding and the second feature embedding with a feed forward artificial neural network.
7. The one or more non-transitory computer-readable media of claim 1 , wherein processing the first feature embedding and the second feature embedding with a third embedding model to generate the three-dimensional mesh model of the object based at least in part on the first feature embedding and the second feature embedding, comprises concatenating the first feature embedding and the second feature embedding.
8. The one or more non-transitory computer-readable media of claim 1 , wherein the loss function comprises a regularization loss, wherein the regularization loss comprises a penalization term for long edges.
9. The one or more non-transitory computer-readable media of claim 1 , wherein the loss function comprises a regularization loss, wherein the regularization loss comprises a penalization term for neighboring faces of the three-dimensional mesh model having different normal directions.
10. The one or more non-transitory computer-readable media of claim 1 , wherein the loss function comprises a regularization loss, wherein the regularization loss comprises a penalization term based on distance from a centroid of neighboring vertices and a vertex.
11. A computer-implemented method comprising:
obtaining first object data, wherein the first object data comprises image data, and second object data, wherein the second object data comprises Light Detection and Ranging (LiDAR) point cloud data;
processing the first object data with a first embedding model to generate a first feature embedding;
processing the second object data with a second embedding model to generate a second feature embedding;
processing the first feature embedding and the second feature embedding with a third embedding model to generate a three-dimensional mesh model of an object based at least in part on the first feature embedding and the second feature embedding;
generating predicted image data based at least in part on the three-dimensional mesh model, wherein the predicted image data comprises a predicted reconstruction of the first object data;
generating predicted point cloud data based at least in part on the three-dimensional mesh model, wherein the predicted point cloud data comprises a predicted reconstruction of the second object data;
evaluating a loss function that compares the predicted image data and the predicted point cloud data with the first object data and the second object data, wherein the loss function comprises a silhouette loss and a Light Detection and Ranging (LiDAR) loss, wherein the silhouette loss compares a generated predicted silhouette based at least in part on the predicted image data against a generated ground truth silhouette based at least in part on the first object data, wherein the LiDAR loss compares vertices of the predicted point cloud data against ground truth vertices data based at least in part on the second object data; and
modifying parameters of at least one of the first embedding model and the second embedding model based on the loss function.
12. The computer-implemented of claim 11 , wherein the first object data comprises data from a plurality of view angles.
13. The computer-implemented of claim 11 , wherein the first object data and the second object data further comprise one or more three-dimensional bounding boxes.
14. The computer-implemented of claim 11 , wherein the first embedding model comprises a fully convolutional neural network.
15. The computer-implemented of claim 11 , wherein the second embedding model comprises a fully convolutional neural network.
16. The computer-implemented of claim 11 , wherein processing the first feature embedding and the second feature embedding with a third embedding model to generate the three-dimensional mesh model of the object based at least in part on the first feature embedding and the second feature embedding, comprises processing the first feature embedding and the second feature embedding with a feed forward artificial neural network.
17. The computer-implemented of claim 1 , wherein processing the first feature embedding and the second feature embedding with a third embedding model to generate the three-dimensional mesh model of the object based at least in part on the first feature embedding and the second feature embedding, comprises concatenating the first feature embedding and the second feature embedding.
18. The computer-implemented of claim 11 , wherein the loss function comprises a regularization loss, wherein the regularization loss comprises a penalization term for long edges.
19. The computer-implemented of claim 11 , wherein the loss function comprises a regularization loss, wherein the regularization loss comprises a penalization term for neighboring faces of the three-dimensional mesh model having different normal directions.
20. A computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining first object data, wherein the first object data comprises image data, and second object data, wherein the second object data comprises Light Detection and Ranging (LiDAR) point cloud data;
processing the first object data with a first embedding model to generate a first feature embedding;
processing the second object data with a second embedding model to generate a second feature embedding;
processing the first feature embedding and the second feature embedding with a third embedding model to generate a three-dimensional mesh model of an object based at least in part on the first feature embedding and the second feature embedding;
generating predicted image data based at least in part on the three-dimensional mesh model, wherein the predicted image data comprises a predicted reconstruction of the first object data;
generating predicted point cloud data based at least in part on the three-dimensional mesh model, wherein the predicted point cloud data comprises a predicted reconstruction of the second object data;
evaluating a loss function that compares the predicted image data and the predicted point cloud data with the first object data and the second object data, wherein the loss function comprises a silhouette loss and a Light Detection and Ranging (LiDAR) loss, wherein the silhouette loss compares a generated predicted silhouette based at least in part on the predicted image data against a generated ground truth silhouette based at least in part on the first object data, wherein the LiDAR loss compares vertices of the predicted point cloud data against ground truth vertices data based at least in part on the second object data; and
modifying parameters of at least one of the first embedding model and the second embedding model based on the loss function.