Fast personalized image generation using a machine learning model

1 . A method of generating images using a machine learning model, comprising:

generating contextualized features based on an object image and a text prompt by a first sub-model of the machine learning model, wherein the object image comprises a target object, and wherein the text prompt depicts a target image and indicates at least one of changing textures of the target object or placing the target object in a particular context;

generating self-attention features from the object image, wherein the self-attention features contain detailed information about the target object;

generating the target image based on the contextualized features and the self-attention features by a second sub-model of the machine learning model, wherein the target image preserves an identity of the target object, and wherein the second sub-model comprises context cross attention layers and object cross attention layers;

wherein the method further comprises:

collecting and transferring the self-attention features into the object cross-attention layers of the second sub-model; and

applying a masking mechanism during a self-attention feature transfer such that only features of a foreground in the object image are injected into a foreground of a generated image while a background of the generated image remains unaffected by the self-attention feature transfer.

2 . The method of claim 1 , further comprising:

generating the object image by replacing a background of a reference image with white color while remaining the target object in the reference image.

3 . The method of claim 1 , wherein the generating contextualized features based on an object image and a text prompt by a first sub-model of the machine learning model comprises:

extracting image features from the object image;

editing the image features based on the text prompt; and

generating a sequence of embedding based on the edited image features.

4 . The method of claim 3 , further comprising:

integrating the sequence of embedding into the context cross-attention layers of the second sub-model.

5 . The method of claim 1 , further comprising:

training the machine learning model using a two-staged training pipeline.

6 . The method of claim 5 , further comprising:

training the first sub-model to compose image features of target objects with text prompts and output prompt-contextualized image embeddings.

7 . The method of claim 6 , further comprising:

training and optimizing the context cross-attention layers and the object cross-attention layers of the second sub-model.

8 . A computing system of generating images using a machine learning model, comprising:

at least one processor; and

at least one non-transitory memory communicatively coupled to the at least one processor and comprising computer-readable instructions that upon execution by the at least one processor cause the at least one processor to perform operations comprising:

generating contextualized features based on an object image and a text prompt by a first sub-model of the machine learning model, wherein the object image comprises a target object, and wherein the text prompt depicts a target image and indicates at least one of changing textures of the target object or placing the target object in a particular context;

generating self-attention features from the object image, wherein the self-attention features contain detailed information about the target object;

generating the target image based on the contextualized features and the self-attention features by a second sub-model of the machine learning model, wherein the target image preserves an identity of the target object, and wherein the second sub-model comprises context cross attention layers and object cross attention layers;

wherein the operations further comprise:

collecting and transferring the self-attention features into the object cross-attention layers of the second sub-model; and

applying a masking mechanism during a self-attention feature transfer such that only features of a foreground in the object image are injected into a foreground of a generated image while a background of the generated image remains unaffected by the self-attention feature transfer.

9 . The computing system of claim 8 , the operations further comprising:

generating the object image by replacing a background of a reference image with white color while remaining the target object in the reference image.

10 . The computing system of claim 8 , wherein the generating contextualized features based on an object image and a text prompt by a first sub-model of the machine learning model comprises:

extracting image features from the object image;

editing the image features based on the text prompt; and

generating a sequence of embedding based on the edited image features.

11 . The computing system of claim 8 , the operations further comprising:

training the first sub-model to compose image features of target objects with text prompts and output prompt-contextualized image embeddings; and

training and optimizing the context cross-attention layers and the object cross-attention layers of the second sub-model.

12 . The computing system of claim 8 , the operations further comprising:

training the machine learning model using a two-staged training pipeline.

13 . A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations comprising:

generating contextualized features based on an object image and a text prompt by a first sub-model of the machine learning model, wherein the object image comprises a target object, and wherein the text prompt depicts a target image and indicates at least one of changing textures of the target object or placing the target object in a particular context;

generating self-attention features from the object image, wherein the self-attention features contain detailed information about the target object;

generating the target image based on the contextualized features and the self-attention features by a second sub-model of the machine learning model, wherein the target image preserves an identity of the target object, and wherein the second sub-model comprises context cross attention layers and object cross attention layers;

wherein the operations further comprise:

collecting and transferring the self-attention features into the object cross-attention layers of the second sub-model; and

applying a masking mechanism during a self-attention feature transfer such that only features of a foreground in the object image are injected into a foreground of a generated image while a background of the generated image remains unaffected by the self-attention feature transfer.

14 . The non-transitory computer-readable storage medium of claim 13 , wherein the generating contextualized features based on an object image and a text prompt by a first sub-model of the machine learning model comprises:

extracting image features from the object image;

editing the image features based on the text prompt; and

generating a sequence of embedding based on the edited image features.

15 . The non-transitory computer-readable storage medium of claim 14 , the operations further comprising:

integrating the sequence of embedding into the context cross-attention layers of the second sub-model.

16 . The non-transitory computer-readable storage medium of claim 13 , the operations further comprising:

training the first sub-model to compose image features of target objects with text prompts and output prompt-contextualized image embeddings; and

training and optimizing the context cross-attention layers and the object cross-attention layers of the second sub-model.

17 . The non-transitory computer-readable storage medium of claim 13 , the operations further comprising:

generating the object image by replacing a background of a reference image with white color while remaining the target object in the reference image.

18 . The non-transitory computer-readable storage medium of claim 13 , the operations further comprising:

training the machine learning model using a two-staged training pipeline.