Text to color palette generation using diffusion models
A method, apparatus, non-transitory computer readable medium, and system for text-to-color palette generation include encoding a text prompt to obtain text embedding. A color embedding is generated based on the text embedding by performing a diffusion process. Then a color palette is generated based on the color embedding. The color palette includes a plurality of colors corresponding to the text prompt.
1 . A method comprising:
encoding, using a text encoder, a text prompt to obtain a text embedding and a plurality of text encodings;
generating, using a machine learning model, a color embedding based on the text embedding by performing an attention mechanism based on the text embedding and the plurality of text encodings and performing a diffusion process based on an output of the attention mechanism; and
generating a color palette, using the machine learning model, based on the color embedding, wherein the color palette includes a plurality of colors corresponding to the text prompt.
2 . The method of claim 1 ,
wherein the text embedding and the color embedding are represented in a same vector space.
3 . The method of claim 1 , wherein generating the color embedding comprises:
sampling a noisy color embedding;
predicting noise based on the text embedding; and
removing the noise from the noisy color embedding to obtain the color embedding.
4 . The method of claim 1 , further comprising:
sampling a plurality of noisy color embeddings; and
generating, using the machine learning model, a plurality of color embeddings based on the plurality of noisy color embeddings.
5 . The method of claim 1 , wherein:
the text prompt includes a base color term and a color modifier.
6 . The method of claim 1 , wherein generating the color palette comprises:
identifying a highest value among a plurality of values of the color embedding; and
selecting a color corresponding to the highest value.
7 . The method of claim 1 , wherein generating the color palette comprises:
filtering the color embedding to obtain a filtered color embedding; and
sorting a plurality of values in the filtered color embedding, wherein the color palette is based on the sorting.
8 . The method of claim 1 , wherein:
the machine learning model is trained to generate color embeddings based on text prompts using a training set including a training text prompt and a ground-truth color embedding.
9 . A method comprising:
initializing a machine learning model;
obtaining a training set including a training text prompt and a ground-truth color embedding; and
training, using the training set, the machine learning model to generate color embeddings based on text prompts by performing a diffusion process on a noisy color embedding, wherein the training comprises:
computing a diffusion loss based on the ground-truth color embedding; and
updating parameters of the machine learning model based on the diffusion loss.
10 . The method of claim 9 , wherein the training comprises:
encoding the training text prompt to obtain a text embedding, wherein the machine learning model takes the text embedding as input.
11 . The method of claim 10 , further comprising:
generating a predicted color embedding based on the noisy color embedding and the text embedding;
computing a loss function by comparing the predicted color embedding to the ground-truth color embedding; and
updating parameters of the machine learning model based on the loss function.
12 . The method of claim 9 , wherein the training comprises:
identifying a pre-determined number of time steps; and
updating parameters of the machine learning model based on the pre-determined number of time steps.
13 . The method of claim 9 , further comprising:
obtaining an image including colors in the ground-truth color embedding, wherein the training text prompt comprises a caption for the image and wherein the ground-truth color embedding is generated based on the image.
14 . An apparatus comprising:
at least one processor;
at least one memory including instructions executable by the at least one processor; and
a machine learning model comprising parameters in the at least one memory, wherein the machine learning model comprises:
a text encoder trained to encode a text prompt to obtain a text embedding and a plurality of text encodings;
a conversion model trained to generate a color embedding based on the text embedding by performing an attention mechanism based on the text embedding and the plurality of text encodings and performing a diffusion process based on an output of the attention mechanism; and
a color palette component configured to generate a color palette based on the color embedding, wherein the color palette includes a plurality of colors corresponding to the text prompt.
15 . The apparatus of claim 14 , wherein:
the conversion model comprises a transformer architecture.
16 . The apparatus of claim 14 , wherein:
the conversion model comprises a diffusion model.
17 . The apparatus of claim 14 , wherein:
the text encoder comprises a transformer architecture.
18 . The apparatus of claim 14 , further comprising:
a data preparation component configured to obtain a training set including a training text prompt and a ground-truth color embedding.