Encoding image values through attribute conditioning

1 . A method comprising:

obtaining a text prompt and a numerical value of a conditioning attribute, wherein the text prompt describes an element, and wherein the numerical value indicates a target level of the conditioning attribute, and wherein the conditioning attribute comprises a continuous attribute;

encoding, using a text encoder, the text prompt to obtain a text embedding;

encoding, using an attribute encoder different from the text encoder, the numerical value of the conditioning attribute to obtain an attribute embedding in a same embedding space as the text embedding, wherein the attribute encoder is trained to encode at least three values of the conditioning attribute;

combining the text embedding and the attribute embedding to obtain a combined embedding representing the element and the target level of the conditioning attribute; and

generating, using an image generation model and based on the combined embedding, a synthesized image having the target level of the conditioning attribute, wherein the synthesized image depicts the element of the text prompt.

2 . The method of claim 1 , further comprising:

applying an adapter network to the combined embedding to obtain an adapted embedding, wherein the synthesized image is generated based on the adapted embedding.

3 . The method of claim 1 , wherein:

the conditioning attribute comprises a lighting attribute.

4 . The method of claim 1 , wherein:

the conditioning attribute comprises camera view information.

5 . The method of claim 1 , wherein generating the synthesized image comprises:

performing a diffusion process on a noise input.

6 . The method of claim 1 , wherein:

the attribute encoder is trained using an image and a ground-truth conditioning attribute corresponding to the image.

7 . The method of claim 1 , further comprising:

obtaining a plurality of conditioning attributes; and

encoding each of the plurality of conditioning attributes with a different attribute encoder to obtain a plurality of attribute embeddings, wherein the synthesized image is generated based on the plurality of attribute embeddings.

8 . A method comprising:

obtaining a text prompt and a numerical value of a conditioning attribute, wherein the text prompt describes an element, and wherein the numerical value indicates a target level of the conditioning attribute, and wherein the conditioning attribute comprises a continuous attribute;

encoding, using a text encoder, the text prompt to obtain a text embedding;

encoding, using an attribute encoder different from the text encoder, the numerical value of the conditioning attribute to obtain an attribute embedding in a same embedding space as the text embedding, wherein the attribute encoder is trained to encode at least three values of the conditioning attribute;

combining the text embedding and the attribute embedding to obtain a combined embedding representing the element and the target level of the conditioning attribute;

generating, using an image generation model and based on the combined embedding, a synthesized image having the target level of the conditioning attribute, wherein the synthesized image depicts the element of the text prompt; and

training, using the synthesized image, the attribute encoder to generate attribute embeddings representing the conditioning attribute for the image generation model.

9 . The method of claim 8 , further comprising:

obtaining a training set including a training image; and

identifying camera view information corresponding to the training image, wherein the conditioning attribute includes the camera view information.

10 . The method of claim 8 , further comprising:

obtaining a training set including a training image, wherein the text prompt describes the training image, and

wherein the image generation model is trained to generate the synthesized image based on the text embedding.

11 . The method of claim 10 , further comprising:

training an adapter network to generate an adapted embedding based on the text embedding and the attribute embedding, wherein the image generation model is trained to generate the synthesized image based on the adapted embedding.

12 . The method of claim 8 , further comprising:

obtaining a training set including a training image; and

training, using the training set, a plurality of different attribute encoders to generate a plurality of different attribute embeddings for the image generation model based on a plurality of different conditioning attributes, respectively.

13 . An apparatus comprising:

at least one processor;

at least one memory including instructions executable by the at least one processor; and

a machine learning model comprising parameters in the at least one memory, wherein the machine learning model configured to perform operations comprising:

obtaining a text prompt and a numerical value of a conditioning attribute, wherein the text prompt describes an element, and wherein the numerical value indicates a target level of the conditioning attribute, and wherein the conditioning attribute comprises a continuous attribute;

encoding, using a text encoder, the text prompt to obtain a text embedding;

encoding, using an attribute encoder different from the text encoder, the numerical value of the conditioning attribute to obtain an attribute embedding in a same embedding space as the text embedding, wherein the attribute encoder is trained to encode at least three values of the conditioning attribute;

combining the text embedding and the attribute embedding to obtain a combined embedding representing the element and the target level of the conditioning attribute; and

generating, using an image generation model and based on the combined embedding, a synthesized image having the target level of the conditioning attribute, wherein the synthesized image depicts the element of the text prompt.

14 . The apparatus of claim 13 , further comprising:

an adapter network trained to generate an adapted embedding based on the text embedding and the attribute embedding, wherein the synthesized image is generated based on the adapted embedding.

15 . The apparatus of claim 13 , wherein:

the text encoder includes a transformer architecture.

16 . The apparatus of claim 13 , wherein:

the attribute encoder includes a position encoder layer and a projection layer.

17 . The apparatus of claim 13 , wherein:

the image generation model comprises a diffusion model.

18 . The apparatus of claim 13 , wherein:

the image generation model comprises a U-Net architecture.

19 . The apparatus of claim 13 , further comprising:

a plurality of different attribute encoders configured to generate a plurality of different attribute embeddings for the image generation model based on a plurality of different conditioning attributes, respectively.