Systems and methods for personalized image generation
Embodiments described herein provide a framework designed to enable personalized image generation capabilities in a pretrained text-to-image generation model. The architecture comprises two replicas of the pretrained text-to-image model—a reference UNet dedicated to extracting visual features from reference images and a base UNet for the actual image generation process. The reference UNet processes reference images to collect the features before each Self-Attention (SA) layer of the reference UNet. The base UNet's SA layers are modified to “Reference Self-Attention” (RSA) layers that allow conditioning on extra features. Using the collected reference features as input, the base UNet equipped with the RSA layers estimates the noise in the input to guide the image generation towards the reference objects.
1 . A method of subject-driven image generation, the method comprising:
receiving, via a data interface, an image containing a subject and a text prompt describing the subject in a visual environment;
encoding, via a first neural network based image model, the image containing the subject into a first latent feature vector;
encoding, via a second neural network based image model, a noisy image into a second latent feature vector based on the text prompt;
receiving, via the data interface, a second image containing the subject;
encoding, via the first neural network based image model, the second image into a third latent feature vector;
generating a combined vector combining values of the first latent feature vector and the third latent feature vector;
generating, via a self-attention layer of the second neural network based image model, an attention score vector based on inputs from the combined vector and the second latent feature vector; and
generating, by the second neural network based image model, an output image based on the attention score vector and the text prompt.
2 . The method of claim 1 , wherein the generating the combined vector includes generating a weighted average of the first latent feature vector and the third latent feature vector.
3 . The method of claim 1 , further comprising:
updating parameters of the first neural network based image model based on the output image while refraining from updating parameters of the second neural network based image model.
4 . The method of claim 1 , wherein the encoding the image containing the subject into a first latent feature vector includes replacing a background of the image with a featureless background.
5 . The method of claim 1 , further comprising:
generating, via a plurality of self-attention layers of the second neural network based image model, a plurality of attention score vectors based on a plurality of latent feature vectors generated by the first neural network based image model,
wherein the generating the output image is further based on the plurality of attention score vectors.
6 . The method of claim 5 , wherein the generating the output image is further based on a plurality of attention score vectors generated via a second plurality of self-attention layers without an input generated by the first neural network based image model.
7 . A system for subject-driven image generation, the system comprising:
a memory that stores a first neural network based image model, a second neural network based image model, and a plurality of processor executable instructions;
a communication interface that receives an image containing a subject and a text prompt describing the subject in a visual environment; and
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
encoding, via a first neural network based image model, the image containing the subject into a first latent feature vector;
encoding, via a second neural network based image model, a noisy image into a second latent feature vector based on the text prompt;
receiving, via the data interface, a second image containing the subject;
encoding, via the first neural network based image model, the second image into a third latent feature vector;
generating a combined vector combining values of the first latent feature vector and the third latent feature vector;
generating, via a self-attention layer of the second neural network based image model, an attention score vector based on inputs from the combined vector and the second latent feature vector; and
generating, by the second neural network based image model, an output image based on the attention score vector and the text prompt.
8 . The system of claim 7 , wherein the generating the combined vector includes generating a weighted average of the first latent feature vector and the third latent feature vector.
9 . The system of claim 7 , the operations further comprising:
updating parameters of the first neural network based image model based on the output image while refraining from updating parameters of the second neural network based image model.
10 . The system of claim 7 , wherein the encoding the image containing the subject into a first latent feature vector includes replacing a background of the image with a featureless background.
11 . The system of claim 7 , the operations further comprising:
generating, via a plurality of self-attention layers of the second neural network based image model, a plurality of attention score vectors based on a plurality of latent feature vectors generated by the first neural network based image model,
wherein the generating the output image is further based on the plurality of attention score vectors.
12 . The system of claim 11 , wherein the generating the output image is further based on a plurality of attention score vectors generated via a second plurality of self-attention layers without an input generated by the first neural network based image model.
13 . A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
receiving, via a data interface, an image containing a subject and a text prompt describing the subject in a visual environment;
encoding, via a first neural network based image model, the image containing the subject into a first latent feature vector;
encoding, via a second neural network based image model, a noisy image into second latent feature vector based on the text prompt;
receiving, via the data interface, a second image containing the subject;
encoding, via the first neural network based image model, the second image into a third latent feature vector;
generating a combined vector combining values of the first latent feature vector and the third latent feature vector;
generating, via a self-attention layer of the second neural network based image model, an attention score vector based on inputs from the combined vector and the second latent feature vector; and
generating, by the second neural network based image model, an output image based on the attention score vector and the text prompt.
14 . The non-transitory machine-readable medium of claim 13 , wherein the generating the combined vector includes generating a weighted average of the first latent feature vector and the third latent feature vector.
15 . The non-transitory machine-readable medium of claim 13 , the operations further comprising:
updating parameters of the first neural network based image model based on the output image while refraining from updating parameters of the second neural network based image model.
16 . The non-transitory machine-readable medium of claim 13 , wherein the encoding the image containing the subject into a first latent feature vector includes replacing a background of the image with a featureless background.
17 . The non-transitory machine-readable medium of claim 13 , the operations further comprising:
generating, via a plurality of self-attention layers of the second neural network based image model, a plurality of attention score vectors based on a plurality of latent feature vectors generated by the first neural network based image model,
wherein the generating the output image is further based on the plurality of attention score vectors.