Compositional image generation and manipulation
Methods and systems for image generation include generating a latent representation of an image, modifying the latent representation of the image based on a trained attribute classifier and a specified attribute input, and decoding the modified latent representation to generate an output image that matches the specified attribute input.
1 . A computer-implemented method for image generation, comprising:
generating a latent representation of an image;
modifying the latent representation of the image based on a trained attribute classifier and a specified attribute input as:
z
0
=
z
ˆ
+
1
γ
0
∑
i
=
1
n
α
0
i
w
i
where {circumflex over (z)} is the latent representation, γ 0 is the specified attribute,
a
0
i
is a scalar value, and w i is a gradient of a log-linear condition predictor, and n is a number of attributes; and
decoding the modified latent representation to generate an output image that matches the specified attribute input.
2 . The method of claim 1 , wherein modifying the latent representation includes applying a plurality of trained classifiers, in accordance with a plurality of specified attribute inputs.
3 . The method of claim 2 , wherein the plurality of specified attribute inputs include conjunction and/or negation relations.
4 . The method of claim 2 , wherein modifying the latent representation includes adding linear combinations of weight vectors from the plurality of attribute classifiers to the latent representation of the image.
5 . The method of claim 1 , wherein modifying the latent representation includes applying classifier guidance to a diffusion model.
6 . The method of claim 5 , wherein applying the classifier guidance includes perturbing unconditioned output of the diffusion model with an output of the attribute classifier.
7 . The method of claim 1 , wherein generating the latent representation includes applying a pretrained encoder model to the input image and wherein decoding the modified latent representation includes applying a pretrained decoder model, corresponding to the pretrained encoder model, to the modified latent representation.
8 . The method of claim 7 , wherein the encoder model and the decoder model are part of a latent diffusion model.
9 . The method of claim 1 , wherein modifying the latent representation includes applying a plurality of different attributes that specify respective quantifiable or natural language inputs.
10 . The method of claim 1 , wherein the log-linear condition predictor is log p(y i |z t ), the log of the conditional probability of attribute y i given a state z t .
11 . A system for image generation, comprising:
a hardware processor; and
a memory that stores a computer program which, when executed by the hardware processor, causes the hardware processor to:
generate a latent representation of an image;
modify the latent representation of the image based on a trained attribute classifier and a specified attribute input as:
z
0
=
z
ˆ
+
1
γ
0
∑
i
=
1
n
α
0
i
w
i
where {circumflex over (z)} is the latent representation, γ 0 is the specified attribute,
a
0
i
is a scalar value, and w i is a gradient of a log linear condition predictor, and n is a number of attributes; and
decode the modified latent representation to generate an output image that matches the specified attribute input.
12 . The system of claim 11 , wherein the computer program further causes the hardware processor to modify the latent representation includes applying a plurality of trained classifiers, in accordance with a plurality of specified attribute inputs.
13 . The system of claim 12 , wherein the plurality of specified attribute inputs include conjunction and/or negation relations.
14 . The system of claim 12 , wherein the computer program further causes the hardware processor to add linear combinations of weight vectors from the plurality of attribute classifiers to the latent representation of the image.
15 . The system of claim 11 , wherein the computer program further causes the hardware processor to apply classifier guidance to a diffusion model.
16 . The system of claim 15 , wherein the computer program further causes the hardware processor to perturb unconditioned output of the diffusion model with an output of the attribute classifier.
17 . The system of claim 11 , wherein the computer program further causes the hardware processor to apply a pretrained encoder model to the input image and to apply a pretrained decoder model, corresponding to the pretrained encoder model, to the modified latent representation.
18 . The system of claim 17 , wherein the encoder model and the decoder model are part of a latent diffusion model.
19 . The system of claim 11 , the computer program further causes the hardware processor to apply a plurality of different attributes that specify respective quantifiable or natural language inputs.