Prior guided latent diffusion
Systems and methods for image processing are described. Embodiments of the present disclosure obtain a text prompt for text guided image generation. A multi-modal encoder of an image processing apparatus encodes the text prompt to obtain a text embedding. A diffusion prior model of the image processing apparatus converts the text embedding to an image embedding. A latent diffusion model of the image processing apparatus generates an image based on the image embedding, wherein the image includes an element described by the text prompt.
1 . A method comprising:
obtaining a text prompt;
encoding the text prompt to obtain a text embedding;
converting the text embedding to an image embedding space using a diffusion prior model by performing diffusion based denoising to predict a denoised image embedding; and
generating an image based on the denoised image embedding using a latent diffusion model (LDM) different from the diffusion prior model, wherein the LDM receives input noise and the denoised image embedding as input and provides the image as output, and wherein the image includes an element described by the text prompt.
2 . The method of claim 1 , further comprising:
generating a plurality of image embeddings based on the text embedding, wherein the denoised image embedding is selected from the plurality of image embeddings.
3 . The method of claim 2 , further comprising:
generating a plurality of images based on the plurality of image embeddings, respectively, wherein each of the plurality of images includes the element described by the text prompt.
4 . The method of claim 2 , further comprising:
comparing each of the plurality of image embeddings to the text embedding; and
computing a similarity score based on the comparison, wherein the denoised image embedding is selected based on the similarity score.
5 . The method of claim 1 , further comprising:
generating a plurality of images based on the denoised image embedding, wherein each of the plurality of images includes the element described by the text prompt.
6 . The method of claim 5 , further comprising:
generating a plurality of noise maps, wherein each of the plurality of images is generated based on one of the plurality of noise maps.
7 . The method of claim 5 , wherein:
the plurality of images has different aspect ratios.
8 . The method of claim 1 , wherein:
the denoised image embedding is in a same embedding space as the text embedding.
9 . The method of claim 1 , further comprising:
generating a modified image embedding using the LDM, wherein the image is generated based on the modified image embedding.
10 . The method of claim 1 , further comprising:
obtaining an image prompt;
encoding the image prompt to obtain an additional image embedding; and
generating an additional image based on the additional image embedding using the LDM, wherein the additional image includes an additional element of the image prompt.
11 . A method comprising:
obtaining training data including a text embedding;
predicting a denoised image embedding based on the text embedding;
computing a loss function based on the denoised image embedding; and
training a diffusion prior model based on the loss function, wherein a latent diffusion model (LDM) receives input noise and a denoised output from the diffusion prior model as input and provides an image as output.
12 . The method of claim 11 , further comprising:
encoding a training image to obtain an image embedding using a multi-modal encoder;
initializing the latent diffusion model (LDM); and
training the LDM to generate a synthetic image based on the image embedding.
13 . The method of claim 12 , further comprising:
obtaining additional training data including an additional image and a text describing the additional image;
encoding the additional image and the text to obtain an additional image embedding and the text embedding respectively, using the multi-modal encoder; and
generating a predicted image embedding using the diffusion prior model based on the additional image embedding.
14 . The method of claim 13 , further comprising:
adding a corresponding noise from a plurality of noise to the additional image embedding to obtain a plurality of additional noise image embeddings;
generating a plurality of predicted image embeddings based on the plurality of additional noise image embeddings;
comparing the plurality of predicted image embeddings to the text embedding; and
selecting the predicted image embedding from the plurality of predicted image embeddings based on the comparison.
15 . The method of claim 13 , wherein:
the image embedding is based on the predicted image embedding.
16 . An apparatus comprising:
at least one processor; and
at least one memory including instructions executable by the at least one processor to perform operations including:
encoding, using a multi-modal encoder, a text prompt to obtain a text embedding;
converting, using a diffusion prior model, the text embedding to an image embedding space by performing diffusion based denoising to predict a denoised image embedding; and
generating, using a latent diffusion model (LDM) different from the diffusion prior model, an image based on the denoised image embedding, wherein the LDM receives input noise and the denoised image embedding as input and provides the image as output, and wherein the image includes an element described by the text prompt.
17 . The apparatus of claim 16 , wherein:
the diffusion prior model comprises a transformer architecture.
18 . The apparatus of claim 16 , wherein:
the LDM comprises a U-Net architecture.
19 . The apparatus of claim 16 , wherein:
the LDM comprises an image decoder.
20 . The apparatus of claim 16 , further comprising:
a training component configured to train the LDM.