IP Library › Granted Patent US 12,430,830
Granted Patent B2
US 12,430,830 · App. 18/228,614 · Granted Sep 30, 2025

Prompt-to-prompt image editing with cross-attention control

Inventors: Kfir Aberman (San Mateo, CA); Amir Hertz (Tel Aviv, IL); Yael Pritch Knaan (Tel Aviv, IL); Ron Mokady (Tel Aviv, IL); Jay Tenenbaum (Tel Aviv, IL); Daniel Cohen-Or (Tel Aviv, IL)
Assignee: GOOGLE LLC
G06T11/60G06F3/04845G06F40/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,430,830
App. No.
18/228,614
Granted
Sep 30, 2025
Kind
B2
Abstract

Some implementations are directed to editing a source image, where the source image is one generated based on processing a source natural language (NL) prompt using a Large-scale language-image (LLI) model. Those implementations edit the source image based on user interface input that indicates an edit to the source NL prompt, and optionally independent of any user interface input that specifies a mask in the source image and/or independent of any other user interface input. Some implementations of the present disclosure are additionally or alternatively directed to applying prompt-to-prompt editing techniques to editing a source image that is one generated based on a real image, and that approximates the real image.

Claims (51)

1. A method implemented by one or more processors, the method comprising:

identifying source cross-attention maps that were produced using cross-attention layers, of a large-scale language-image (LLI) model, in generating a source image based on processing a source natural language (NL) prompt using the LLI model;

identifying one or more source random seeds that were utilized in generating the source image based on processing the source NL prompt using the LLI model;

subsequent to generating the source image:

receiving user interface input that indicates an edit to the source NL prompt that was used in generating the source image;

in response to receiving the user interface input that indicates the edit to the source NL prompt:

generating, in multiple iterations of processing using the LLI model, an edited image that is visually similar to the source image but that includes visual modifications consistent with the edit, to the NL prompt, indicated by the user interface input, wherein generating, in the iterations of processing using the LLI model, the edited image, comprises:

processing, in the iterations of processing using the LLI model:

one or more features generated based on the edit to the source NL prompt, and

the source random seeds, and

injecting, in at least some of the iterations of generating the edited image using the LLI model, at least a portion of the source cross-attention maps.

2. The method of claim 1 , wherein the edit comprises a replacement, of a subset of tokens of the source NL prompt, with one or more replacement tokens that differ from the subset of tokens of the source NL prompt.

3. The method of claim 2 , wherein the one or more features generated based on the edit to the source NL prompt comprise a text embedding of a modified prompt that conforms to the source NL prompt, but replaces the subset of tokens of the source NL prompt with the one or more replacement tokens.

4. The method of claim 3 , wherein injecting, in the at least some of the iterations of generating the edited image using the LLI model, at least a portion of the source cross-attention maps comprises:

using the entirety of the source cross-attention maps in processing the text embedding, of the modified prompt, in the at least some of the iterations.

5. The method of claim 4 , wherein the at least some of the iterations are a subset of the iterations and wherein in other iterations, that are not included in the subset of the iterations, other cross-attention maps are utilized in processing the text embedding and the source cross-attention maps are not utilized in processing the text embedding.

6. The method of claim 5 , wherein the subset of the iterations is an initial continuous sequence of the iterations.

7. The method of claim 5 , wherein the subset of the iterations comprises more than five percent of the iterations, but less than ninety-five percent of the iterations.

8. The method of claim 1 , wherein the edit comprises an addition, of one or more additional tokens, to the source NL prompt.

9. The method of claim 8 , wherein the one or more features generated based on the edit to the source NL prompt comprise a text embedding of a modified prompt that includes the source NL prompt and the additional tokens.

10. The method of claim 9 , wherein injecting, in at least some of the iterations of generating the edited image using the LLI model, at least a portion of the source cross-attention maps comprises:

using the entirety of the source cross-attention maps in processing a portion of the text embedding that corresponds to the source NL prompt, wherein the source cross-attention maps are not utilized in processing an additional portion of the text embedding that corresponds to the additional tokens.

11. The method of claim 10 , wherein the at least some of the iterations are a subset of the iterations and wherein in other iterations, that are not included in the subset, the source cross-attention maps are not utilized in processing the portion of the text embedding that corresponds to the source NL prompt.

12. The method of claim 11 , wherein the subset of the iterations comprises more than five percent of the iterations, but less than ninety-five percent of the iterations.

13. The method of claim 1 , wherein the edit comprises an adjustment of emphasis on one or more emphasis tokens of the source tokens of the source NL prompt, the adjustment of emphasis being an increase or decrease of emphasis.

14. The method of claim 13 , wherein the one or more features generated based on the edit to the source NL prompt comprise one or more scaled attention maps for the one or more emphasis tokens, and further comprising:

identifying an emphasis portion, of the source cross-attention maps, that corresponds to the one or more emphasis tokens; and

generating the one or more scaled attention maps by scaling the emphasis portion in correspondence with the adjustment of emphasis.

15. The method of claim 14 , wherein the adjustment of emphasis is the increase of emphasis and wherein generating the one or more scaled attention maps by scaling the emphasis portion in correspondence with the adjustment of emphasis comprises:

increasing values of the emphasis portion by a factor.

16. The method of claim 15 , wherein the increase of emphasis, indicated by the user interface input, is of a particular magnitude that is one of multiple candidate degrees of magnitude and wherein the factor is proportional to the particular magnitude.

17. The method of claim 14 , wherein a text embedding of the source NL prompt is processed in the iterations of processing using the LLI model, and wherein the text embedding comprises an emphasis embedding portion corresponding to the one or more emphasis tokens and a remaining portion embedding corresponding to a remainder of the source NL prompt after excluding the emphasis portion.

18. The method of claim 17 , wherein the at least a portion of the source cross-attention maps is a remaining portion of the source cross-attention maps after excluding the emphasis portion and wherein injecting, in the at least some of the iterations of generating the edited image using the LLI model, the at least a portion of the source cross-attention maps comprises:

using, in the at least some of the iterations, the remaining portion of the source cross-attention maps in processing the remaining portion embedding, wherein the one or more scaled source cross-attention maps are utilized in processing the emphasis embedding portion in the at least some of the iterations.

19. The method of claim 1 , further comprising:

generating the source image based on processing the source natural language (NL) prompt using the LLI model.

20. A method implemented by one or more processors, the method comprising:

identifying a real image captured by a real camera;

identifying a natural language (NL) caption for the real image;

generating, using an inversion process and based on the real image, a noise vector for the real image;

processing, using a large-scale language-image (LLI) model and the noise vector, the NL caption to generate a source image that approximates the real image;

identifying source cross-attention maps that were produced using cross-attention layers, of the LLI model, in generating the source image;

identifying source random seeds that were utilized in generating the source image;

subsequent to generating the source image:

receiving user interface input that indicates an edit to the NL caption that was used in generating the source image;

in response to receiving the user interface input that indicates the edit to the NL caption:

generating, in multiple iterations of processing using the LLI model, an edited image that is visually similar to the source image but includes visual modifications consistent with the edit, to the NL caption, indicated by the user interface input, wherein generating, in the multiple iterations of processing using the LLI model, the edited image, comprises:

processing, in the iterations of processing using the LLI model:

one or more features generated based on the edit to the source NL caption, and

the source random seeds, and

injecting, in at least some of the iterations of generating the edited image using the LLI model, at least a portion of the source cross-attention maps.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 31, 2024
From: ABERMAN, KFIR; HERTZ, AMIR; PRITCH KNAAN, YAEL; MOKADY, RON; TENENBAUM, JAY; COHEN-OR, DANIEL
To: GOOGLE LLC
Reel/Frame 066311/0029 →
Continuity (2)
Provisional Application 63394187 · Aug 1, 2022
Related Publication 20240037822A1 · Feb 1, 2024
References Cited (63)
US 20230095092A1 · Xiao · 2023 [cited by examiner]
US 20230377226A1 · Saharia · 2023 [cited by examiner]
US 20240005604A1 · Kreis · 2024 [cited by examiner]
US 20240135611A1 · Costin · 2024 [cited by examiner]
WO WO2023225344A1 · 2023 [cited by examiner]
Wang, Jianan, et al. “Manitrans: Entity-level text-guided image manipulation via token-wise semantic alignment and generation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Jun. 202… [cited by examiner]
Wu, Chen Henry, and Fernando De la Torre. “Making Text-to-Image Diffusion Models Zero-Shot Image-to-Image Editors by Inferring “Random Seeds”.” NeurIPS 2022 Workshop on Score-Based Methods, Nov. 2022, pp. 1-7. (Year: 20… [cited by examiner]
Brock, A. et al., “Large Scale GAN Training For High Fidelity Natural Image Synthesis;” Cornell University, arXiv.org. arXiv:1809.11096v1; 29 pages; Sep. 28, 2018. [cited by applicant]
Goodfellow, I. et al., “Generative Adversarial Nets;” In Advances in Neural Information Processing Systems; 9 pages; dated 2014. [cited by applicant]
Radford et al., “Learning Transferable Visual Models From Natural Language Supervision” arXiv:2103.00020v1 [cs.CV] 48 pages, dated Feb. 26, 2021. [cited by applicant]
Vaswani, A. et al. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30 (NIPS 2017), 11 pages; dated 2017. [cited by applicant]
Abdal, R. et al., “Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?”; In Proceedings of the IEEE/CVF International Conference on Computer Vision; pp. 4432-4441; dated 2019. [cited by applicant]
Abdal, R. et al., “CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions”; arXiv.org; arXiv:2112.05219v1; 17 pages; dated Dec. 9, 2021. [cited by applicant]
Alaluf, Y. et al., “HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing”; In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; pp. 18511-18521; dated 2022. [cited by applicant]
Avrahami, O. et al., “Blended Latent Diffusion”; arXiv.org; arXiv:2206.02779v1; 20 pages; dated Jun. 6, 2022. [cited by applicant]
Avrahami, Omri et al., “Blended diffusion for text-driven editing of natural images”; In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; pp. 18208-18218; dated 2022. [cited by applicant]
Bar-Tal, O. et al., “Text2LIVE: Text-Driven Layered Image and Video Editing”; arXiv.org; arXiv:2204.02491v2; 21 pages; dated May 25, 2022. [cited by applicant]
Bau, D. et al., “Paint by Word”; arXiv.org; arXiv:2103.10951v2; 10 pages; dated Mar. 24, 2021. [cited by applicant]
Crowson, K. et al., “VQGAN-Clip: Open Domain Image Generation and Editing with Natural Language Guidance”; arXiv.org; arXiv:2204.08583v2; 32 pages; dated Sep. 4, 2022. [cited by applicant]
Dhariwal, P. et al., “Diffusion Models Beat GANs on Image Synthesis”; Advances in Neural Information Processing Systems; 15 pages; dated 2021. [cited by applicant]
Ding, M. et al., “CogView: Mastering Text-to-Image Generation via Transformers”; Advances in Neural Information Processing Systems; 14 pages; dated 2021. [cited by applicant]
Esser, P. et al., “Taming Transformers for High-Resolution Image Synthesis”; In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; pp. 12873-12883; dated 2021. [cited by applicant]
Gafni, O. et al., “Make-A-Scene: Scene-Based Text-To-Image Generation with Human Priors”; arXiv.org; arXiv:2203.13131v1; 17 pages; dated Mar. 24, 2022. [cited by applicant]
Gal, R. et al, “StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators”; arXiv.org; arXiv:2108.00946v2; dated Dec. 16, 2021. [cited by applicant]
Hinz, T. et al., “Semantic Object Accuracy for Generative Text-To-Image Synthesis”; IEEE Transactions on Pattern Analysis and Machine Intelligence; dated 2020. [cited by applicant]
Ho, J. et al., “Denoising Diffusion Probabilistic Models”; Advances in Neural Information Processing Systems; 12 pages; dated 2020. [cited by applicant]
Ho, J. et al., “Classifier-Free Diffusion Guidance”; In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications; 8 pages; dated 2021. [cited by applicant]
Karras, T. et al., “Alias-Free Generative Adversarial Networks”; Advances in Neural Information Processing Systems; 12 pages; dated 2021. [cited by applicant]
Karras, T. et al., “A Style-Based Generator Architecture for Generative Adversarial Networks”; In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; pp. 4401-4410; dated 2019. [cited by applicant]
Karras, T. et al, “Analyzing and Improving the Image Quality of StyleGAN”; In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; pp. 8110-8119; dated 2020. [cited by applicant]
Kim, G. et al, “DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation”; In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; pp. 2426-2435; dated 2022. [cited by applicant]
Kwon, G. et al, “CLIPstyler: Image Style Transfer with a Single Text Condition”; arXiv.org; arXiv:2112.00374v1; 19 pages; dated Dec. 1, 2021. [cited by applicant]
Lample, G. et al., “Fader Networks: Manipulating Images by Sliding Attributes”; Advances in Neural Information Processing Systems; 10 pages; dated 2017. [cited by applicant]
Li, B. et al, “Controllable Text-to-Image Generation”; Advances in Neural Information Processing Systems; 11 pages; dated 2019. [cited by applicant]
Li, W. et al; “Object-Driven Text-To-Image Synthesis Via Adversarial Training”; In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; pp. 12174-12182; dated 2019. [cited by applicant]
Mokady, R. et al., “Self-Distilled StyleGAN: Towards Generation from Internet Photos”; In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings; 9 pages; dated 2022. [cited by applicant]
Nichol, A. et al., “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models”; arXiv.org; arXiv:2112.10741v2; 20 pages; dated Dec. 2021. [cited by applicant]
Patashnik, O. et al., “StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery”; arXiv.org; arXiv:2103.17249v1; 18 pages; dated Mar. 31, 2021. [cited by applicant]
Qiao. T. et al., “Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge”; Advances in Neural Information Processing Systems; 11 pages; dated 2019. [cited by applicant]
Qiao, T. et al., “MirrorGAN: Learning Text-to-image Generation by Redescription”; In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; pp. 1505-1514; dated 2019. [cited by applicant]
Ramesh, A. et al., “Hierarchical Text-Conditional Image Generation with CLIP Latents”; arXiv.org; arXiv:2204.06125v1; 27 pages; dated Apr. 2022. [cited by applicant]
Ramesh, A. et al., “Zero-Shot Text-to-Image Generation”; In International Conference on Machine Learning, PMLR; 11 pages; dated 2021. [cited by applicant]
Roich, D. et al., “Pivotal Tuning for Latent-based editing of Real Images”; ACM Transactions on Graphics (TOG); 14 pages; dated 2022. [cited by applicant]
Rombach, R. et al., “High-Resolution Image Synthesis with Latent Diffusion Models”; arXiv.org; arXiv:2112.10752v1; 45 pages; dated Dec. 2021. [cited by applicant]
Ronneberger, O. et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation”; In International Conference on Medical Image Computing and Computer-Assisted Intervention; Springer; pp. 234-241; dated 2015. [cited by applicant]
Sohl-Dickstein, J. et al., “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics”; In International Conference on Machine Learning: 10 pages; dated 2015. [cited by applicant]
Song, J. et al., “Denoising Diffusion Implicit Models”; In International Conference on Learning Representations; 19 pages; dated 2020. [cited by applicant]
Song, Y. et al., “Generative Modeling by Estimating Gradients of the Data Distribution”; Advances in Neural Information Processing Systems; 13 pages; dated 2019. [cited by applicant]
Tao, M. et al., “DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image Synthesis”; arXiv.org, Cornell University; arXiv:2008.05865v1; 13 pages; dated Aug. 2020. [cited by applicant]
Tov, O. et al., “Designing an Encoder for StyleGAN Image Manipulation”; arXiv.org, Cornell University; arXiv:2102.02766v1; 33 pages; dated Feb. 2021. [cited by applicant]
Wang, T. et al., “High-Fidelity GAN Inversion for Image Attribute Editing”; arXiv.org, Cornell University; arXiv:2109.06590; 22 pages; dated 2021. [cited by applicant]
Xia, W. et al., “TediGAN: Text-Guided Diverse Face Image Generation and Manipulation”; In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; pp. 2256-2265; dated 2021. [cited by applicant]
Xia, W. et al., GAN Inversion: A Survey; arXiv.org, Cornell University; arXiv:2101.05278v4; 21 pages; dated Aug. 2021. [cited by applicant]
Yu, J. et al, “Scaling Autoregressive Models for Content-Rich Text-to-Image Generation”; arXiv.org, Cornell University; arXiv:2206.10789v1; 49 pages; dated Jun. 2022. [cited by applicant]
Zhang, Z. et al., “Photographic Text-To-Image Synthesis with a Hierarchically-nested Adversarial Network”; In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; pp. 6199-6208; dated 2018. [cited by applicant]
Zhu, J. et al., “In-Domain GAN Inversion for Real Image Editing”; arXiv.org, Cornell University; arXiv:2004.00049; 31 pages; dated Jul. 2020. [cited by applicant]
Zhu, J-Y. et al., “Generative Visual Manipulation on the Natural Image Manifold”; In European Conference on Computer Vision; Springer; pp. 597-613; dated 2016. [cited by applicant]
Saharia et al., “Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding” arXiv:2205.11487v1 [cs.CV] 46 pages, dated May 23, 2022. [cited by applicant]
Hertz, A. et al., “Prompt-to-Prompt Image Editing with Cross Attention Control”; arXiv.org, Cornell University; arXiv:2208.01626v1; 19 pages; dated Aug. 2, 2022. [cited by applicant]
Nichol, A. et al., “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models”; arXiv.org; Cornell University; arXiv:2112.10741v3; 20 pages; dated Mar. 8, 2022. [cited by applicant]
Li, B. et al., “ManiGAN: Text-Guided Image Manipulation”; IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 10 pages; dated Jun. 13, 2020. [cited by applicant]
Bar-Tal, O. et al., “Text2LIVE: Text-Driven Layered Image and Video Editing”; arXiv.org; Cornell University; 21 pages; dated Apr. 5, 2022. [cited by applicant]
European Patent Office; International Search Report and Written Opinion issued in Application No. PCT/US2023/029149; 13 pages; dated Nov. 16, 2023. [cited by applicant]
Cited By (1)
US 12,711,678