Text to color palette generation using diffusion models

1 . A method comprising:

encoding, using a text encoder, a text prompt to obtain a text embedding and a plurality of text encodings;

generating, using a machine learning model, a color embedding based on the text embedding by performing an attention mechanism based on the text embedding and the plurality of text encodings and performing a diffusion process based on an output of the attention mechanism; and

generating a color palette, using the machine learning model, based on the color embedding, wherein the color palette includes a plurality of colors corresponding to the text prompt.

2 . The method of claim 1 ,

wherein the text embedding and the color embedding are represented in a same vector space.

3 . The method of claim 1 , wherein generating the color embedding comprises:

sampling a noisy color embedding;

predicting noise based on the text embedding; and

removing the noise from the noisy color embedding to obtain the color embedding.

4 . The method of claim 1 , further comprising:

sampling a plurality of noisy color embeddings; and

generating, using the machine learning model, a plurality of color embeddings based on the plurality of noisy color embeddings.

5 . The method of claim 1 , wherein:

the text prompt includes a base color term and a color modifier.

6 . The method of claim 1 , wherein generating the color palette comprises:

identifying a highest value among a plurality of values of the color embedding; and

selecting a color corresponding to the highest value.

7 . The method of claim 1 , wherein generating the color palette comprises:

filtering the color embedding to obtain a filtered color embedding; and

sorting a plurality of values in the filtered color embedding, wherein the color palette is based on the sorting.

8 . The method of claim 1 , wherein:

the machine learning model is trained to generate color embeddings based on text prompts using a training set including a training text prompt and a ground-truth color embedding.

9 . A method comprising:

initializing a machine learning model;

obtaining a training set including a training text prompt and a ground-truth color embedding; and

training, using the training set, the machine learning model to generate color embeddings based on text prompts by performing a diffusion process on a noisy color embedding, wherein the training comprises:

computing a diffusion loss based on the ground-truth color embedding; and

updating parameters of the machine learning model based on the diffusion loss.

10 . The method of claim 9 , wherein the training comprises:

encoding the training text prompt to obtain a text embedding, wherein the machine learning model takes the text embedding as input.

11 . The method of claim 10 , further comprising:

generating a predicted color embedding based on the noisy color embedding and the text embedding;

computing a loss function by comparing the predicted color embedding to the ground-truth color embedding; and

updating parameters of the machine learning model based on the loss function.

12 . The method of claim 9 , wherein the training comprises:

identifying a pre-determined number of time steps; and

updating parameters of the machine learning model based on the pre-determined number of time steps.

13 . The method of claim 9 , further comprising:

obtaining an image including colors in the ground-truth color embedding, wherein the training text prompt comprises a caption for the image and wherein the ground-truth color embedding is generated based on the image.

14 . An apparatus comprising:

at least one processor;

at least one memory including instructions executable by the at least one processor; and

a machine learning model comprising parameters in the at least one memory, wherein the machine learning model comprises:

a text encoder trained to encode a text prompt to obtain a text embedding and a plurality of text encodings;

a conversion model trained to generate a color embedding based on the text embedding by performing an attention mechanism based on the text embedding and the plurality of text encodings and performing a diffusion process based on an output of the attention mechanism; and

a color palette component configured to generate a color palette based on the color embedding, wherein the color palette includes a plurality of colors corresponding to the text prompt.

15 . The apparatus of claim 14 , wherein:

the conversion model comprises a transformer architecture.

16 . The apparatus of claim 14 , wherein:

the conversion model comprises a diffusion model.

17 . The apparatus of claim 14 , wherein:

the text encoder comprises a transformer architecture.

18 . The apparatus of claim 14 , further comprising:

a data preparation component configured to obtain a training set including a training text prompt and a ground-truth color embedding.