Generating perception data using a critic network
Techniques for improving synthetic training data generation by models used to generate training data for training object detection models are disclosed. Synthetic data may be generated by a synthetic training data generation model and provided to a crit model. The critic network may determine, based on real-world data associated with similar scenarios represented by the generated synthetic data, whether the generated synthetic data is distinguishable from real-world data. If so, the system may adjust the parameters of the synthetic training data generation model and again execute the model to generate synthetic data. This subsequent synthetic data is then critiqued by the critic network. This process may be iteratively performed until the synthetic data generated by the synthetic training data generation model is indistinguishable from real-world data. The synthetic training data generation model may then be used to generate data that may be used to train other models.
1 . A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving real-world image data associated with an environment;
determining, based at least in part on the real-world image data, a prompt associated with image data represented by a subset of the real-world image data;
executing a diffusion model, using the prompt as first input, to generate first synthetic image data as first output;
executing a discriminatory network, using the first synthetic image data as second input, to generate first loss data associated with the first synthetic image data;
backpropagating the first loss data through the diffusion model to generate a modified diffusion model;
executing the modified diffusion model, using the prompt as third input, to generate second synthetic image data as second output;
executing the discriminatory network, using the second synthetic image data as fourth input, to generate second loss data associated with the second synthetic image data; and
based at least in part on determining that a loss value of the second loss data falls below a threshold, executing, based at least in part on the second loss data, the modified diffusion model to generate synthetic image training data to train one or more detection models.
2 . The system of claim 1 , wherein determining the prompt comprises:
determining one or more captions for images of the subset of the real-world image data; and
determining the prompt based at least in part on the one or more captions.
3 . The system of claim 1 , wherein:
the subset of the real-world image data comprises a cluster of images of the real-world image data; and
the operations further comprise:
determining one or more captions for images of the real-world image data;
clustering a subset of the images of the real-world image data associated with a caption of the one or more captions into the cluster of images; and
determining the prompt based at least in part on the caption.
4 . The system of claim 1 , wherein determining the prompt comprises:
determining a type of object represented in an image of the subset of the real-world image data; and
determining the prompt based at least in part on the type of object.
5 . The system of claim 1 , wherein:
executing the modified diffusion model to generate the second synthetic image data as second output comprises:
iteratively executing the modified diffusion model to generate subsequent synthetic image data;
executing the discriminatory network to generate subsequent synthetic data detection information;
modifying one or more parameters of the modified diffusion model based at least in part on the subsequent synthetic data detection information; and
executing the modified diffusion model comprising the one or more modified parameters to generate the second synthetic image data as second output.
6 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, perform operations comprising:
executing a diffusion model, using a prompt as first input, to generate first synthetic training data as first output;
executing a critic, using the first synthetic training data as second input, to generate first loss data associated with the first synthetic training data as second output;
backpropagating the first loss data through the diffusion model to generate a modified diffusion model;
executing the modified diffusion model, using the prompt as third input, to generate second synthetic training data as third output;
executing the critic, using the second synthetic training data as fourth input, to generate second loss data associated with the second synthetic training data; and
based at least in part on determining that a loss value of the second loss data falls below a threshold, executing the modified diffusion model based at least in part on the second loss data to generate third synthetic training data as fourth output for training one or more detection models.
7 . The one or more non-transitory computer-readable media of claim 6 , wherein the operations further comprise:
determining a classification for one or more data units of real-world data; and
determining the prompt based at last in part on the classification.
8 . The one or more non-transitory computer-readable media of claim 6 , wherein the prompt comprises an indication of a modality.
9 . The one or more non-transitory computer-readable media of claim 6 , wherein the operations further comprise:
using real-world image data and corresponding real-world lidar data to generate depth image data; and
determining the prompt based at last in part on the depth image data.
10 . The one or more non-transitory computer-readable media of claim 6 , wherein:
the first loss data comprises an indication that the first synthetic training data is synthetic; and
executing the modified diffusion model to generate the second synthetic training data as the third output comprises configuring an object mask at the modified diffusion model.
11 . The one or more non-transitory computer-readable media of claim 6 , wherein the operations further comprise determining the prompt based on a caption associated with a cluster of data units of real-world data.
12 . The one or more non-transitory computer-readable media of claim 6 , wherein the operations further comprise determining the prompt based on a type of object.
13 . The one or more non-transitory computer-readable media of claim 6 , wherein the operations further comprise:
training an object detection model using the third synthetic training data; and
providing the object detection model to a vehicle computing system for use in controlling a vehicle in an environment.
14 . The one or more non-transitory computer-readable media of claim 6 , wherein the prompt comprises one or more of:
an image;
lidar data;
radar data;
audio data; or
machine-readable data.
15 . A method comprising:
executing a diffusion model, using a prompt as first input, to generate first synthetic data as first output;
executing a critic, using the first synthetic data as second input, to generate first loss data associated with the first synthetic data as second output;
backpropagating the first loss data through the diffusion model to generate an updated diffusion model;
executing the updated diffusion model, using the prompt as third input, to generate second synthetic data as third output;
executing the critic, using the second synthetic data as fourth input, to generate second loss data associated with the second synthetic data; and
based at least in part on determining that a loss value of the second loss data falls below a threshold, executing the updated diffusion model to generate third synthetic data as fourth output for training one or more detection models.
16 . The method of claim 15 , wherein:
executing the critic further generates synthetic data detection information; and
the method further comprises modifying conditioning data of the diffusion model based at least in part on the synthetic data detection information.
17 . The method of claim 15 , further comprising:
determining one or more object labels for objects represented in real-world data; and
determining the prompt based at least in part on the one or more object labels.
18 . The method of claim 17 , further comprising:
captioning data units of real-world data;
clustering the data units based at least in part on captions associated with the data units to determine clustered data units and associated cluster data; and
determining the prompt based on the cluster data.
19 . The method of claim 15 , wherein backpropagating the first loss data through the diffusion model comprises:
determining that a second loss value represented in the first loss data meets or exceeds the threshold; and
based at least on part on determining that the second loss value meets or exceeds the threshold, backpropagating the first loss data through the diffusion model to generate the updated diffusion model.
20 . The method of claim 15 , further comprising:
determining a classification for one or more data units of real-world image data;
determining that a quantity of the one or more data units is below a threshold quantity; and
determining the prompt based at least in part on the classification.