Machine learning systems for generating output images based on style elements of input images

1 . A system comprising:

one or more non-transitory memories storing computer-executable instructions; and

one or more hardware processors to execute the computer-executable instructions to:

provide a plurality of image embeddings, a first weight parameter, and a second weight parameter as inputs to a diffusion model that is trained to determine output images having characteristics based on image embeddings, wherein the diffusion model generates an output image by:

performing a first denoising process based on an input image to generate a first interim output;

performing a first cross-attention process based on the plurality of image embeddings, the first weight parameter, and the first interim output to generate a second interim output, wherein the first weight parameter is used to control a first correspondence between the second interim output and one or more style characteristics represented by the plurality of image embeddings;

performing a second denoising process based on the second interim output to generate a third interim output; and

performing a second cross-attention process based on the plurality of image embeddings, the second weight parameter, and the third interim output to generate a fourth interim output, wherein the second weight parameter is used to control a second correspondence between the one or more style characteristics and the fourth interim output, and wherein the second correspondence is greater than the first correspondence;

wherein the output image is determined based at least in part on the fourth interim output, and wherein the diffusion model further generates the output image by:

determining input indicative of a type of object;

determining a portion of the plurality of image embeddings that represents an object that corresponds to the type of object; and

disregarding the portion of the plurality of image embeddings that represents the type of object during one or more of the first denoising process, the second denoising process, the first cross-attention process, or the second cross-attention process.

2 . The system of claim 1 , further comprising computer-executable instructions to:

determine first image data that represents an image having the one or more style characteristics;

encode the first image data using an image encoder to generate the plurality of image embeddings; and

generate the input image by:

determining a single embedding based on the first image data; and

providing the single embedding as an input to the diffusion model, wherein the diffusion model performs a third denoising process based on the single embedding to generate the input image.

3 . The system of claim 2 , further comprising computer-executable instructions to:

determine edge data based on characteristics of pixels of the input image, wherein the edge data is indicative of one or more boundaries between at least a first region of the input image having a first characteristic and a second region of the input image having a second characteristic; and

provide the edge data as an input to the diffusion model for generation of the output image, wherein the output image includes at least a subset of the one or more boundaries.

4 . The system of claim 1 , further comprising computer-executable instructions to:

receive first text data indicative of one or more content characteristics;

encode the first text data to generate a text embedding; and

provide the text embedding as an input to the diffusion model, wherein the diffusion model further generates the output image by:

performing a third cross-attention process based on the text embedding and the first interim output to generate a fifth interim output, wherein the first cross-attention process determines the second interim output based on the fifth interim output; and

performing a fourth cross-attention process based on the text embedding and the third interim output to generate a sixth interim output, wherein the second cross-attention process determines the fourth interim output based on the sixth interim output.

5 . The system of claim 4 , wherein the diffusion model further generates the output image by:

determining the input indicative of the type of object based on the text embedding, wherein the type of object is indicated in the first text data.

6 . The system of claim 4 , wherein the diffusion model comprises first parameters associated with the first denoising process and the second denoising process, second parameters associated with the first cross-attention process and the second cross-attention process, and third parameters associated with the third cross-attention process and the fourth cross-attention process, the system further comprising computer-executable instructions to:

train the diffusion model to generate output images having characteristics based on image embeddings by:

preventing modification of the first parameters and the third parameters; and

permitting modification of the second parameters.

7 . The system of claim 4 , further comprising computer-executable instructions to:

determine first image data that represents an image having the one or more style characteristics;

encode the first image data using an image encoder to generate the plurality of image embeddings;

wherein:

the diffusion model comprises first parameters associated with the first denoising process and the second denoising process, second parameters associated with the first cross-attention process and the second cross-attention process, and third parameters associated with the third cross-attention process and the fourth cross-attention process, and

the image encoder comprises fourth parameters associated with generation of image embeddings based on image data; and

train the image encoder to generate image embeddings based on image data by:

preventing modification of the first parameters, the second parameters, and the third parameters; and

permitting modification of the fourth parameters.

8 . The system of claim 1 , further comprising computer-executable instructions to:

provide query data comprising a plurality of queries as an input to the diffusion model, wherein the first cross-attention process uses at least a first subset of the plurality of queries to determine the second interim output, and the second cross-attention process uses at least a second subset of the plurality of queries to determine the fourth interim output.

9 . A system comprising:

one or more non-transitory memories storing computer-executable instructions; and

one or more hardware processors to execute the computer-executable instructions to:

provide an input image, a plurality of image embeddings, a first weight parameter, and a second weight parameter as inputs to a first machine learning model that is trained to determine output images having characteristics based on image embeddings, wherein the first machine learning model:

performs a first denoising process based on the input image to generate a third output;

performs a first cross-attention process based on the plurality of image embeddings and the first weight parameter, wherein a first correspondence between the plurality of image embeddings and a first output of the first cross-attention process is based at least in part on the first weight parameter and the third output;

performs a second denoising process on the first output to generate a fourth output;

performs a second cross-attention process based on the plurality of image embeddings, the first output, and the second weight parameter, wherein a second correspondence between the plurality of image embeddings and a second output of the second cross-attention process is based at least in part on the second weight parameter and the fourth output, and wherein the second correspondence is greater than the first correspondence; and

determines an output image based at least in part on the second output.

10 . The system of claim 9 , further comprising computer-executable instructions to:

determine a text embedding based on first text data; and

provide the text embedding as an input to the first machine learning model, wherein the first machine learning model further:

performs a third cross-attention process based on the text embedding and the third output to generate a fifth output, wherein the first cross-attention process determines the first output further based on the fifth output; and

performs a fourth cross-attention process based on the text embedding and the fourth output to generate a sixth output, wherein the second cross-attention process determines the second output further based on the sixth output.

11 . The system of claim 10 , wherein the first machine learning model further:

determines, based on the text embedding, a first characteristic of an object indicated in the first text data; and

determines a portion of the plurality of image embeddings associated with one or more second characteristics that correspond to the first characteristic;

wherein the first output and the second output are determined independent of the portion of the plurality of image embeddings.

12 . The system of claim 9 , wherein the plurality of image embeddings is determined based on first image data, the system further comprising computer-executable instructions to:

generate the input image by:

determining a single embedding based on the first image data; and

providing the single embedding as an input to one of the first machine learning model or a second machine learning model, wherein the one of the first machine learning model or the second machine learning model performs a denoising process based on the single embedding to generate the input image.

13 . The system of claim 12 , further comprising computer-executable instructions to:

determine edge data based on the input image, wherein the edge data is indicative of one or more boundaries between at least a first region of the input image having a first characteristic and a second region of the input image having a second characteristic; and

provide the edge data as an input to the first machine learning model, wherein the output image includes at least a subset of the one or more boundaries.

14 . The system of claim 9 , further comprising computer-executable instructions to:

provide input indicative of a first characteristic of an object to the first machine learning model, wherein:

the first machine learning model further determines, based on the plurality of image embeddings, a portion of the plurality of image embeddings associated with one or more second characteristics that correspond to the first characteristic; and

the first output and the second output are determined independent of the portion of the plurality of image embeddings.

15 . The system of claim 9 , further comprising computer-executable instructions to:

provide query data comprising a plurality of queries as an input to the first machine learning model, wherein the first cross-attention process uses at least a first subset of the plurality of queries to determine the first output, and the second cross-attention process uses at least a second subset of the plurality of queries to determine the second output.

16 . A system comprising:

one or more non-transitory memories storing computer-executable instructions; and

one or more hardware processors to execute the computer-executable instructions to:

determine a single embedding based on first image data;

provide the single embedding as an input to a first machine learning model, wherein the first machine learning model performs a denoising process based on the single embedding to generate an input image; and

provide the input image, a plurality of image embeddings, a first weight parameter, and a second weight parameter as inputs to one or more of the first machine learning model or a second machine learning model, wherein the one or more of the first machine learning model or the second machine learning model is trained to determine output images having characteristics based on image embeddings, and wherein the one or more of the first machine learning model or the second machine learning model:

performs a first cross-attention process based on the plurality of image embeddings and the first weight parameter, wherein a first correspondence between the plurality of image embeddings and a first output of the first cross-attention process is based at least in part on the first weight parameter;

performs a second cross-attention process based on the plurality of image embeddings, the first output, and the second weight parameter, wherein a second correspondence between the plurality of image embeddings and a second output of the second cross-attention process is based at least in part on the second weight parameter, and wherein the second correspondence is greater than the first correspondence; and

determines an output image based at least in part on the second output.

17 . The system of claim 16 , further comprising computer-executable instructions to:

provide input indicative of a first characteristic of an object to the one or more of the first machine learning model or the second machine learning model, wherein:

the one or more of the first machine learning model or the second machine learning model further determines, based on the plurality of image embeddings, a portion of the plurality of image embeddings associated with one or more second characteristics that correspond to the first characteristic; and

the first output and the second output are determined independent of the portion of the plurality of image embeddings.

18 . The system of claim 16 , wherein the one or more of the first machine learning model or the second machine learning model further:

performs a first denoising process based on the input image to generate a third output, wherein the first cross-attention process determines the first output further based at least in part on the third output; and

performs a second denoising process based on the first output to generate a fourth output, wherein the second cross-attention process determines the second output based at least in part on the fourth output.

19 . The system of claim 18 , further comprising computer-executable instructions to:

determine a text embedding based on first text data; and

provide the text embedding as an input to the one or more of the first machine learning model or the second machine learning model, wherein the one or more of the first machine learning model or the second machine learning model further:

performs a third cross-attention process based on the text embedding and the third output to generate a fifth output, wherein the first cross-attention process determines the first output further based on the fifth output;

performs a fourth cross-attention process based on the text embedding and the fourth output to generate a sixth output, wherein the second cross-attention process determines the second output further based on the sixth output;

determines, based on the text embedding, a first characteristic of an object indicated in the first text data; and

determines a portion of the plurality of image embeddings associated with one or more second characteristics that correspond to the first characteristic;

wherein the first output and the second output are determined independent of the portion of the plurality of image embeddings.

20 . The system of claim 16 , further comprising computer-executable instructions to:

determine edge data based on the input image, wherein the edge data is indicative of one or more boundaries between at least a first region of the input image having a first characteristic and a second region of the input image having a second characteristic; and

provide the edge data as an input to the one or more of the first machine learning model or the second machine learning model, wherein the output image includes at least a subset of the one or more boundaries.