Systems and methods for text-to-image generation using language models
Embodiments described herein provide a mechanism for replacing existing text encoders in text-to-image generation models with more powerful pre-trained language models. Specifically, a translation network is trained to map features from the pre-trained language model output into the space of the target text encoder. The training preserves the rich structure of the pre-trained language model while allowing it to operate within the text-to-image generation model. The resulting modularized text-to-image model receives prompt and generates an image representing the features contained in the prompt.
1 . A method of integrating a pre-trained language model into a text-to-image model of a target text encoder and an image generator model, the method comprising:
receiving, via a data interface, a text prompt describing image content;
encoding, via a pre-trained language model, the text prompt into a source embedding in a source representation space;
encoding, via the target text encoder, the prompt into a target embedding in a target representation space;
transforming, via a translation network encoder, the source embedding from the source representation space into a transformed source embedding in the target representation space;
generating, via a translation network decoder, a decoded source embedding in the source representation space from the transformed source embedding;
computing a first loss based, at least in part, on a difference between the target embedding and the transformed source embedding;
computing a second loss based, at least in part, on a difference between the source embedding and the decoded source embedding;
training the translation network encoder and the translation network decoder based on the first loss and the second loss; and
building a modularized text-to-image model by replacing the target text encoder in the text-to-image model with the pre-trained language model followed by the trained translation network encoder.
2 . The method of claim 1 , wherein the training comprises updating the parameters of the translation network encoder and the translation network decoder via propagation through the translation network encoder, the translation network decoder, and the pre-trained language model based on the first loss and the second loss while keeping the pre-trained language model frozen.
3 . The method of claim 1 , further comprising:
generating, by the image generator model, an image from the transformed source embedding.
4 . The method of claim 1 , further comprising:
receiving, via a data interface, an image associated with the text prompt;
generating, via a conditioning network, a condition source embedding from the transformed source embedding;
generating, by an image encoder, an image feature embedding in a latent image space from the image;
generating a noisy image embedding from the image feature embedding by adding gaussian noise to the image feature embedding;
generating a noisy image-text embedding by concatenating the condition source embedding and the noisy image embedding;
generating, by a denoising network, a noise-reduced image-text embedding from the noisy image-text embedding;
computing a third loss, based at least in part, on an unscaled gaussian noise and the noise-reduced image-text embedding; and
updating parameters of the denoising network and conditioning network based on the third loss.
5 . The method of claim 4 , wherein the updating comprises updating the parameters of the denoising network and the conditioning network while keeping the pre-trained language model frozen.
6 . The method of claim 1 , further comprising:
generating, by a discriminator network, a target distribution over the target representation space and a transformed source distribution over the target representation space from the target embedding and the transformed source embedding, respectively;
computing a third loss based, at least in part, on the target distribution and the transformed source distribution; and
updating parameters of the discriminator network, the translation network encoder, and the translation network decoder based on the first loss, second loss, and third loss via backpropagation.
7 . The method of claim 6 , wherein the updating comprises updating the parameters of the discriminator network, the translation network encoder, and translation network decoder while keeping the pre-trained language model frozen.
8 . A system for integrating a pre-trained language model into a text-to-image model of a target text encoder and an image generator model, the system comprising:
a communication interface that receives a plurality of training samples;
a memory containing machine readable medium storing machine executable code;
one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to:
receiving, via a data interface, a text prompt describing image content;
encoding, via a pre-trained language model, the text prompt into a source embedding in a source representation space;
encoding, via the target text encoder, the prompt into a target embedding in a target representation space;
transforming, via a translation network encoder, the source embedding from the source representation space into a transformed source embedding in the target representation space;
generating, via a translation network decoder, a decoded source embedding in the source representation space from the transformed source embedding;
computing a first loss based, at least in part, on a difference between the target embedding and the transformed source embedding;
computing a second loss based, at least in part, on a difference between the source embedding and the decoded source embedding;
training the translation network encoder and the translation network decoder based on the first loss and the second loss; and
building a modularized text-to-image model by replacing the target text encoder in the text-to-image model with the pre-trained language model followed by the trained translation network encoder.
9 . The system of claim 8 , wherein the processor is further configured to train the translation network encoder and translation network decoder via propagation through the translation network encoder, the translation network decoder, and the pre-trained language model based on the first loss and the second loss while keeping the pre-trained language model frozen.
10 . The system of claim 8 , wherein the processor is further configured to:
generate, by the image generator model, an image from the transformed source embedding.
11 . The system of claim 8 , wherein the processor is further configured to:
receive, via a data interface, an image associated with the prompt;
generate, via a conditioning network, a condition source embedding from the transformed source embedding;
generate, by an image encoder, an image feature embedding in a latent image space from the image;
generate a noisy image embedding from the image feature embedding by adding gaussian noise to the image feature embedding;
generate a noisy image-text embedding by concatenating the condition source embedding and the noisy image embedding;
generate, by a denoising network, a noise-reduced image-text embedding from the noisy image-text embedding;
compute a third loss, based at least in part, on an unscaled gaussian noise and the noise-reduced image-text embedding; and
update parameters of the denoising network and conditioning network based on the third loss via backpropagation.
12 . The system of claim 11 , wherein to update parameters, the processor is further configured to update the parameters of the denoising network and the conditioning network while keeping the pre-trained language model frozen.
13 . The system of claim 8 , wherein the processor is further configured to:
generate, by a discriminator network, a target distribution over the target representation space and a transformed source distribution over the target representation space from the target embedding and the transformed source embedding, respectively;
compute a third loss based, at least in part, on the target distribution and the transformed source distribution; and
update parameters of the discriminator network, the translation network encoder, and the translation network decoder based on the first loss, second loss, and third loss via backpropagation.
14 . The system of claim 13 , wherein to update parameters, the processor is further configured to update parameters of the discriminator network, the translation network encoder, and the translation network decoder while keeping the pre-trained language model frozen.
15 . A non-transitory machine-readable medium comprising a plurality of machine-executable instructions for integrating a pre-trained language model into a text-to-image model of a target text encoder and an image generator model which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
receiving, via a data interface, a text prompt describing image content;
encoding, via a pre-trained language model, the text prompt into a source embedding in a source representation space;
encoding, via the target text encoder, the prompt into a target embedding in a target representation space;
transforming, via a translation network encoder, the source embedding from the source representation space into a transformed source embedding in the target representation space;
generating, via a translation network decoder, a decoded source embedding in the source representation space from the transformed source embedding;
computing a first loss based, at least in part, on a difference between the target embedding and the transformed source embedding;
computing a second loss based, at least in part, on a difference between the source embedding and the decoded source embedding;
training the translation network encoder and the translation network decoder based on the first loss and the second loss; and
building a modularized text-to-image model by replacing the target text encoder in the text-to-image model with the pre-trained language model followed by the trained translation network encoder.
16 . The non-transitory machine-readable medium of claim 15 , wherein the training comprises updating the parameters of the translation network encoder and the translation network decoder via propagation through the translation network encoder, the translation network decoder, and the pre-trained language model based on the first loss and the second loss while keeping the pre-trained language model frozen.
17 . The non-transitory machine-readable medium of claim 15 , further comprising:
generating, by the image generator model, an image from the transformed source embedding.
18 . The non-transitory machine-readable medium of claim 15 , further comprising:
receiving, via a data interface, an image associated with the prompt;
generating, via a conditioning network, a condition source embedding from the transformed source embedding;
generating, by an image encoder, an image feature embedding in a latent image space from the image;
generating a noisy image embedding from the image feature embedding by adding gaussian noise to the image feature embedding;
generating a noisy image-text embedding by concatenating the condition source embedding and the noisy image embedding;
generating, by a denoising network, a noise-reduced image-text embedding from the noisy image-text embedding;
computing a third loss, based at least in part, on an unscaled gaussian noise and the noise-reduced image-text embedding; and
updating parameters of the denoising network and conditioning network based on the third loss via backpropagation.
19 . The non-transitory machine-readable medium of claim 18 , wherein the updating comprises updating the parameters of the denoising network and the conditioning network while keeping the pre-trained language model frozen.
20 . The non-transitory machine-readable medium of claim 15 , further comprising:
generating, by a discriminator network, a target distribution over the target representation space and a transformed source distribution over the target representation space from the target embedding and the transformed source embedding, respectively;
computing a third loss based, at least in part, on the target distribution and the transformed source distribution; and
updating parameters of the discriminator network, the translation network encoder, and the translation network decoder based on the first loss, second loss, and third loss via backpropagation.