Neural networks for synthetic data generation with discrete and continuous variable features
In various examples, systems and methods are disclosed relating to neural networks for synthetic data generation with discrete and continuous variable features. In training, an encoder can determine a plurality of encodings from a plurality of samples of training data, and the continuous generative model can operate as a decoder that is conditioned on the plurality of encodings to generate an estimated output to update the encoder and the continuous generative model. The discrete generative model can be trained over the plurality of encodings to learn to generate discrete variables corresponding to the distribution of information represented by the training data. At runtime, the discrete generative model can be used to generate a discrete variable from an input prompt, and can provide the discrete variable to the continuous generative model for the continuous generative model to generate an output, such an image, conditioned on the discrete variable.
1 . A processor comprising:
one or more circuits to:
apply an input to a generative model to cause the generative model to generate a conditioner, the input indicative of one or more features for an output, the generative model configured based at least on a plurality of encodings; and
apply each of the input and the conditioner to a decoder to cause the decoder to generate the output based at least on the input and the conditioner, the decoder configured based at least on a subset of the plurality of encodings.
2 . The processor of claim 1 , wherein the one or more circuits are to apply training data to an encoder to cause the encoder to generate the plurality of encodings for configuration of the decoder.
3 . The processor of claim 1 , wherein:
the plurality of encodings comprise a plurality of discrete variables;
the generative model comprises a discrete variable diffusion model to infer the conditioner based at least on the plurality of discrete variables; and
the decoder comprises a continuous variable diffusion model.
4 . The processor of claim 1 , wherein the one or more circuits are to:
determine, by applying a training data sample as input to an encoder, an estimated encoding of the training data sample in a latent space;
determine, by applying the estimated encoding as input to the generative model, an estimated output corresponding to the estimated encoding; and
update the encoder, the decoder, and the generative model based at least on the estimated output and the training data sample.
5 . The processor of claim 4 , wherein:
the encoder is to determine the estimated encoding as an estimated discrete class corresponding to one or more features represented by the training data sample; and
the decoder is to determine the estimated output as a continuous data representation of the one or more features.
6 . The processor of claim 5 , wherein the training data sample is a first training data sample, the estimated output is a first estimated output, and the one or more circuits are to:
cause the decoder to generate a second estimated output based at least on a second training data sample; and
update the decoder based at least on the second estimated output and a second training data sample, either (i) without a second estimated encoding of the second training data sample determined by the encoder or (ii) with an example encoding independent of the second training data sample.
7 . The processor of claim 6 , wherein the one or more circuits are to update the decoder either (i) without the second estimated encoding or (ii) with the example encoding responsive to an encoding drop-out metric corresponding to the second estimated encoding.
8 . The processor of claim 1 , wherein:
the decoder comprises a continuous variable diffusion model; and
the generative model comprises at least one of an autoregressive model or a discrete variable diffusion model.
9 . The processor of claim 1 , wherein the one or more circuits are to apply regularization to at least one of the conditioner or a plurality of encodings used to configure the generative model.
10 . The processor of claim 1 , wherein the processor is comprised in at least one of:
a system for generating synthetic data;
a system for performing simulation operations;
a system for performing conversational AI operations;
a system for performing collaborative content creation for 3D assets;
a system comprising one or more large language models (LLMs);
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
11 . A system comprising:
one or more processing units to execute operations comprising:
applying an input to a generative model to cause the generative model to generate a conditioner, the input indicative of one or more features for an output, the generative model configured based at least on a plurality of encodings; and
applying each of the input and the conditioner to a decoder to cause the decoder to generate the output based at least on the input and the conditioner, the decoder configured based at least on a subset of the plurality of encodings.
12 . The system of claim 11 , wherein the one or more processing units are to apply training data to an encoder to cause the encoder to generate the plurality of encodings for configuration of the decoder.
13 . The system of claim 11 , wherein:
the plurality of encodings comprise a plurality of discrete variables;
the generative model comprises a discrete variable diffusion model to infer the conditioner based at least on the plurality of discrete variables; and
the decoder comprises a continuous variable diffusion model.
14 . The system of claim 11 , wherein the one or more processing units are to:
determine, by applying a training data sample as input to an encoder, an estimated encoding of the training data sample in a latent space;
determine, by applying the estimated encoding as input to the generative model, an estimated output corresponding to the estimated encoding; and
update the encoder, the decoder, and the generative model based at least on the estimated output and the training data sample.
15 . The system of claim 14 , wherein:
the encoder is to determine the estimated encoding as an estimated discrete class corresponding to one or more features represented by the training data sample; and
the decoder is to determine the estimated output as a continuous data representation of the one or more features.
16 . The system of claim 15 , wherein the training data sample is a first training data sample, the estimated output is a first estimated output, and the one or more processing units are to:
cause the decoder to generate a second estimated output based at least on a second training data sample; and
update the decoder based at least on the second estimated output and a second training data sample, either (i) without a second estimated encoding of the second training data sample determined by the encoder or (ii) with an example encoding independent of the second training data sample.
17 . The system of claim 11 , wherein:
the decoder comprises a continuous variable diffusion model; and
the generative model comprises at least one of an autoregressive model or a discrete variable diffusion model.
18 . The system of claim 11 , wherein the system is comprised in at least one of:
a system for generating synthetic data;
a system for performing simulation operations;
a system for performing conversational AI operations;
a system for performing collaborative content creation for 3D assets;
a system comprising one or more large language models (LLMs);
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
19 . A method, comprising:
receiving, by one or more processors, an input indicative of one or more features of an image to generate;
applying, by the one or more processors, the input to a discrete variable generative model and to a continuous variable generative model;
generating, by the discrete variable generative model, at least one discrete variable of a plurality of predefined discrete variables to represent the one or more features;
applying, by the one or more processors, the at least one discrete variable to the continuous variable generative model;
generating, by the continuous variable generative model, and based at least on the at least one discrete variable and the input indicative of the one or more features, the image; and
causing, by a device, at least one of (i) a simulation to be rendered that includes the image or (ii) presentation of the image.
20 . The method of claim 19 , wherein the continuous variable generative model is a first continuous variable generative model, wherein generating the image comprises generating the image further based on output from a second continuous variable generative model that is not conditioned on the at least one discrete variable.