Granted Patent B2

US 12,361,616 · App. 18/169,444 · Granted Jul 15, 2025

Inventors: Nicholas Isaac Kolkin (San Francisco, CA); Elya Shechtman (Seattle, WA)

Assignee: ADOBE INC.

G06T11/60G06T5/70G06T2200/24

Abstract

Claims (52)

1. A method for image generation, comprising:

obtaining an original image depicting an element and a target prompt describing a modification to the element;

computing a first output and a second output using a diffusion model, wherein the first output is based on a description of the element and the second output is based on the target prompt;

computing a difference between the first output and the second output by subtracting the first output from the second output; and

generating a modified image including the modification to the element of the original image based on the difference.

2. The method of claim 1 , further comprising:

obtaining an anchor prompt that includes a first modifier of the element, wherein the description is based on the anchor prompt and the target prompt includes a second modifier that describes the modification to the element.

3. The method of claim 1 , further comprising:

generating the description of the element based on the original image and the target prompt.

4. The method of claim 1 , further comprising:

adding first noise to the original image to obtain a noise image; and

removing second noise from the noise image based on the difference to obtain the modified image.

5. The method of claim 4 , further comprising:

computing a weighted sum of the first output and the difference, wherein the second noise is based on the weighted sum.

6. The method of claim 4 , further comprising:

determining a number of noise addition steps for adding the first noise; and

computing the difference at each of a plurality of noise removal steps for removing the second noise.

7. The method of claim 1 , further comprising:

obtaining a mask indicating a region corresponding to the element of the original image, wherein the modified image is generated based on the mask.

8. The method of claim 7 , further comprising:

generating a noise map based on the original image and the mask, wherein the modified image is generated based on the noise map.

9. The method of claim 1 , further comprising:

encoding an anchor prompt and the target prompt using a text encoder to obtain an encoded anchor prompt and an encoded target prompt, wherein the first output and the second output are based on the encoded anchor prompt and the encoded target prompt, respectively.

10. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to:

obtain an original image depicting an element, an anchor prompt describing the element, and a target prompt describing a modification to the element;

compute a first output based on the anchor prompt and a second output based on the target prompt using a diffusion model;

compute a difference between the first output and the second output by subtracting the first output from the second output; and

generate a modified image including the modification to the element of the original image based on the difference.

11. The non-transitory computer readable medium of claim 10 , wherein the instructions further cause the processor to:

add first noise to the original image to obtain a noise image; and

remove second noise from the noise image based on the difference to obtain the modified image.

12. The non-transitory computer readable medium of claim 11 , wherein the instructions further cause the processor to:

compute a weighted sum of the first output and the difference, wherein the second noise is based on the weighted sum.

13. The non-transitory computer readable medium of claim 11 , wherein the instructions further cause the processor to:

determine a number of noise addition steps for adding the first noise; and

compute the difference at each of a plurality of noise removal steps for removing the second noise.

14. The non-transitory computer readable medium of claim 10 , wherein the instructions further cause the processor to:

obtain a mask indicating a region corresponding to the element of the original image, wherein the modified image is generated based on the mask.

15. The non-transitory computer readable medium of claim 10 , wherein:

the anchor prompt includes a first modifier of the element and the target prompt includes a second modifier of the element, wherein the second modifier describes the modification to the element.

16. A system for image generation, comprising:

one or more processors;

one or more memory components coupled with the one or more processors; and

a diffusion model configured to compute a first output based on an anchor prompt and a second output based on a target prompt, compute a difference between the first output and the second output by subtracting the first output from the second output, and generate a modified image including a modification to an element of an original image based on the difference.

17. The system of claim 16 , further comprising:

a mask generation model configured to generate a mask indicating a location of the element, wherein the modified image is generated based on the mask.

18. The system of claim 16 , further comprising:

a noise component configured to add first noise to the original image to obtain a noise image, wherein the diffusion model is further configured to remove second noise from the noise image based on the difference to obtain the modified image.

19. The system of claim 16 , further comprising:

a user interface configured to obtain the original image and the target prompt.

20. The system of claim 16 , further comprising:

a text encoder configured to encode the anchor prompt and the target prompt to obtain an encoded anchor prompt and an encoded target prompt, wherein the first output and the second output are based on the encoded anchor prompt and the encoded target prompt, respectively.

Assignments (1)

Continuity (2)

References Cited (22)