Multilingual text-to-image generation
Systems and methods for image processing are provided. One aspect of the systems and methods includes obtaining a text prompt in a first language. Another aspect of the systems and methods includes encoding the text prompt using a multilingual encoder to obtain a multilingual text embedding. Yet another aspect of the systems and methods includes processing the multilingual text embedding using a diffusion prior model to obtain an image embedding, wherein the diffusion prior model is trained to process multilingual text embeddings from the first language and a second language based on training data from the first language and the second language. Yet another aspect of the systems and methods includes generating an image using a diffusion model based on the image embedding, wherein the image includes an element corresponding to the text prompt.
1 . A method comprising:
obtaining a text prompt in a first language;
encoding the text prompt using a multilingual encoder to obtain a multilingual text embedding;
processing the multilingual text embedding using a diffusion prior model to obtain an image embedding, wherein the diffusion prior model is trained to perform denoising based on multilingual text embeddings from the first language and a second language to obtain input guidance for an image generation model different from the diffusion prior model, and wherein the diffusion prior model is trained based on training data from the first language and the second language; and
generating an image using the image generation model by performing denoising based on the image embedding, wherein the image includes an element corresponding to the text prompt.
2 . The method of claim 1 , further comprising:
obtaining an additional text prompt in the second language;
encoding the additional text prompt using the multilingual encoder to obtain an additional multilingual text embedding;
processing the additional multilingual text embedding using the diffusion prior model to obtain an additional image embedding; and
generating an additional image using the diffusion model based on the additional image embedding, wherein the additional image includes an additional element corresponding to the additional text prompt.
3 . The method of claim 1 , further comprising:
generating a plurality of intermediate image embeddings corresponding to a plurality of diffusion time steps, wherein the image is generated based on the plurality of intermediate image embeddings.
4 . The method of claim 1 , further comprising:
obtaining a causal attention mask, wherein the image embedding is generated based on the causal attention mask.
5 . The method of claim 1 , further comprising:
generating a plurality of image embeddings using the diffusion prior model;
computing a similarity score between each of the plurality of image embeddings and the multilingual text embedding; and
selecting the image embedding from the plurality of image embeddings based on the similarity score.
6 . The method of claim 1 , wherein:
the image embedding is in a same embedding space as the multilingual text embedding.
7 . A method comprising:
obtaining training data including a plurality of images, a first plurality of image captions in a first language, and a second plurality of image captions in a second language;
encoding the first plurality of image captions and the second plurality of image captions using a multilingual encoder to obtain a plurality of multilingual text embeddings;
processing the plurality of multilingual text embeddings using a diffusion prior model to obtain a plurality of predicted image embeddings corresponding to the first plurality of image captions in the first language and the second plurality of image captions in the second language to obtain input guidance for an image generation model different from the diffusion prior model, and wherein the diffusion prior model is trained based on training data from the first language and the second language; and
training the diffusion prior model to generate image embeddings based on multilingual text embeddings from the first language and the second language, wherein the diffusion prior model is trained based on the plurality of predicted image embeddings and the plurality of images.
8 . The method of claim 7 , further comprising:
identifying a plurality of ground-truth image embeddings corresponding to the plurality of images, respectively; and
comparing the plurality of predicted image embeddings to the plurality of ground-truth image embeddings, wherein the diffusion prior model is trained based on the comparison.
9 . The method of claim 7 , further comprising:
generating a plurality of predicted images based on the plurality of predicted image embeddings using a diffusion model; and
comparing the plurality of predicted images to the plurality of images, respectively, wherein the diffusion prior model is trained based on the comparison.
10 . The method of claim 9 , wherein:
the diffusion model is pretrained prior to training the diffusion prior model.
11 . The method of claim 7 , further comprising:
translating the first plurality of image captions to obtain the second plurality of image captions.
12 . The method of claim 7 , wherein:
the plurality of images includes a first subset of images corresponding to the first language and a second subset of images corresponding to the second language, the first subset of images being different from the second subset of images.
13 . The method of claim 7 , wherein:
the multilingual encoder is pretrained prior to training the diffusion prior model.
14 . A system comprising:
at least one memory component; and
at least one processing device coupled to the at least one memory component, wherein the processing device is configured to execute instructions stored in the at least one memory component to perform operations comprising:
processing, using a diffusion prior model, a multilingual text embedding from a first language to obtain an image embedding, wherein the diffusion prior model is trained to perform denoising based on multilingual text embeddings from the first language and a second language to obtain input guidance for an image generation model different from the diffusion prior model, and wherein the diffusion prior model is trained based on training data from the first language and the second language; and
generating, using the image generation model comprising a diffusion model, an image by performing denoising based on the image embedding, wherein the image includes an element corresponding to the multilingual text embedding.
15 . The system of claim 14 , wherein the at least one processing device is configured to execute instructions stored in the at least one memory component to perform operations comprising:
encoding, using a multilingual encoder, a text prompt in the first language to obtain the multilingual text embedding.
16 . The system of claim 15 , wherein the multilingual encoder comprises a multimodal encoder for text and images.
17 . The system of claim 14 , wherein the at least one processing device is configured to execute instructions stored in the at least one memory component to perform operations comprising:
training the diffusion prior model to generate image embeddings based on multilingual text embeddings from the first language and the second language, wherein the diffusion prior model is trained based on a plurality of predicted images.
18 . The system of claim 14 , wherein:
the image embedding is in a same embedding space as the multilingual text embedding.
19 . The system of claim 14 , wherein the diffusion prior model comprises a transformer architecture.
20 . The system of claim 14 , wherein the diffusion model comprises a UNet architecture.