Refining synthetic data with a generative adversarial network using auxiliary inputs
The present invention extends to methods, systems, and computer program products for refining synthetic data with a Generative Adversarial Network (GAN) using auxiliary inputs. Refined synthetic data can be rendered more realistically than the original synthetic data. Refined synthetic data also retains annotation metadata and labeling metadata used for training of machine learning models. GANs can be extended to use auxiliary channels as inputs to a refiner network to provide hints about increasing the realism of synthetic data. Refinement of synthetic data enhances the use of synthetic data for additional applications.
1. A method comprising:
accessing synthetic image data representing an image of a roadway scene and including ground truth data annotations;
accessing auxiliary data including image segmentation data, depth map data, and object edge data, the image segmentation data segmenting the synthetic image into multiple regions and indicating an object in each of the multiple regions, the depth map differentiating how each of the multiple regions is to appear based on object distance from a camera, the object edge data defining transitions between a plurality of objects in the synthetic image data;
generating refined synthetic image data using the image segmentation data, the depth map data, and the object edge data, as hints, including refining the synthetic image data by applying textures to the synthetic image data considering the transitions between different objects, from among the plurality of objects, in different regions, from among the multiple regions, have different object distances from the camera and without altering the ground truth data annotations; and
outputting the refined synthetic image data.
2. The method of claim 1 , wherein accessing the auxiliary data comprises accessing one or more auxiliary data streams corresponding to the image of the roadway scene.
3. The method of claim 2 , wherein accessing the auxiliary data comprises accessing a pixel level semantic segmentation of the image of the roadway scene.
4. The method of claim 2 , wherein accessing the synthetic image data comprises accessing pixel values for pixels in the image of the roadway scene.
5. The method of claim 1 , wherein accessing the auxiliary data comprises accessing a depth map image and an image segmentation image.
6. The method of claim 1 , wherein accessing the auxiliary data including the image segmentation data comprises accessing image segmentation data indicating one of foliage or a side of a building in a region.
7. A method for refining machine learning model training data, the method comprising:
accessing synthetic image data representing an image of a roadway scene and including annotations annotating the synthetic image data with ground truth data for the roadway scene;
accessing one or more auxiliary data streams corresponding to the image including image segmentation data, depth map data, and object edge data, the image segmentation data segmenting the synthetic image into multiple regions and indicating an object in each of the multiple regions, the depth map differentiating how each of the multiple regions is to appear based on object distance from a camera, the object edge data defining transitions between a plurality of objects in the synthetic image data;
refining the synthetic image data using contents of the image segmentation data, the depth map data, and the object edge data, as hints, including applying correct textures to the synthetic image data considering transitions between different objects, from among the plurality of objects, in different regions, from among the multiple regions, have different object distances from the camera and without altering the annotations; and
outputting the refined synthetic image data representing a refined image of the roadway scene.
8. The method of claim 7 , wherein accessing the synthetic image data representing the image of a roadway scene comprises accessing previously refined synthetic image data representing the image of the roadway scene;
further comprising receiving one or more feedback parameters associated with a discriminator decision classifying the previously refined synthetic data; and
wherein refining the synthetic image data comprises using the one or more feedback parameters to further refine the previously refined synthetic image data without altering the annotations.
9. The method of claim 7 , wherein accessing the one or more auxiliary data streams corresponding to the image of the roadway scene comprises accessing a pixel level semantic segmentation of the image of the roadway scene.
10. The method of claim 7 , wherein accessing the one or more auxiliary data streams corresponding to the image of the roadway scene comprises accessing the depth map data that defines varying levels of detail for objects based on distance of the objects from a camera.
11. The method of claim 7 , wherein accessing the synthetic image data comprises accessing pixel values for pixels in the image of the roadway scene.
12. The method of claim 7 , further comprising extracting an auxiliary data stream from other image data.
13. The method of claim 12 , wherein extracting the auxiliary data stream from the other image data comprises extracting the auxiliary data stream from a sensor that is synchronized with a camera data stream.
14. The method of claim 7 , further comprising using the refined synthetic image data to train a machine learning module associated with autonomous driving of a vehicle.
15. The method of claim 7 , wherein accessing the one or more auxiliary data streams comprises accessing image segmentation data indicating one of foliage or a side of a building in a region.
16. A computer system comprising:
system memory storing instructions; and
one or more processors executing the instructions stored in the system memory to perform the following:
access synthetic image data representing an image of a roadway scene and including annotations annotating the synthetic image data with ground truth data for the roadway scene;
access auxiliary data streams corresponding to the image including image segmentation data, depth map data, and object edge data, the image segmentation data segmenting the synthetic image into multiple regions and indicating an object in each of the multiple regions, the depth map differentiating how each of the multiple regions is to appear based on object distance from a camera, the object edge data defining transitions between different objects in the synthetic image data;
refine the synthetic image data use contents of the image segmentation data, the depth map data, and the object edge data, as hints, including applying textures to the synthetic image data considering transitions between different objects in different regions, from among the multiple regions, have different object distances from the camera and without altering the annotations; and
output the refined synthetic image data.
17. The computer system of claim 16 , wherein the instructions configured to access the synthetic image data representing the image of the roadway scene comprise instructions configured to access previously refined synthetic image data representing the image of the roadway scene;
further comprising instructions configured to receive feedback parameters associated with a discriminator decision classifying the previously refined synthetic data; and
wherein the instructions configured to refine the synthetic image data comprise instructions configured to use the feedback parameters to further refine the previously refined synthetic image data without altering the annotations.
18. The computer system of claim 16 , further comprising instructions configured to extract an auxiliary data stream, from among the auxiliary data streams, from a sensor that is synchronized with a camera data stream.
19. The computer system of claim 16 , further comprising instructions configured to use the refined synthetic image data to train a machine learning module associated with autonomous driving of a vehicle.
20. The computer system of claim 16 , wherein the instructions configured to access the auxiliary data streams comprise instructions configured to access image segmentation data indicating one of foliage or a side of a building in a region.