IP Library › Granted Patent US 12,536,626
Granted Patent B2
US 12,536,626 · App. 18/307,546 · Granted Jan 27, 2026

Mask-robust image inpainting utilizing machine-learning models

Inventors: Sohrab Amirghodsi (Seattle, WA); Lingzhi Zhang (Philadelphia, PA); Connelly Barnes (Seattle, WA); Elya Shechtman (Seattle, WA); Yuqian Zhou (Urbana, IL); Zhe Lin (Fremont, CA)
Assignee: Adobe Inc.
G06T5/77G06T5/30G06T5/50G06T7/11G06T2207/20076G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,536,626
App. No.
18/307,546
Granted
Jan 27, 2026
Kind
B2
Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for inpainting digital images utilizing mask-robust machine-learning models. In particular, in one or more embodiments, the disclosed systems obtain an initial mask for an object depicted in a digital image. Additionally, in some embodiments, the disclosed systems generate, utilizing a mask-robust inpainting machine-learning model, an inpainted image from the digital image and the initial mask. Moreover, in some implementations, the disclosed systems generate a relaxed mask that expands the initial mask. Furthermore, in some embodiments, the disclosed systems generate a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask.

Claims (83)

1 . A system comprising:

one or more memory devices comprising a digital image and an initial mask for an object portrayed in the digital image; and

one or more processors coupled to the one or more memory devices that cause the system to train a mask-robust inpainting machine-learning model by:

generating, utilizing the mask-robust inpainting machine-learning model, an inpainted image from the digital image, wherein the inpainted image comprises modified pixels inside and outside the initial mask;

generating, utilizing an additional inpainting machine-learning model, a pseudo-ground-truth inpainted image from the digital image utilizing a dilated mask generated from the initial mask;

determining a measure of loss by comparing the inpainted image and the pseudo-ground-truth inpainted image; and

tuning parameters of the mask-robust inpainting machine-learning model based on the measure of loss.

2 . The system of claim 1 , wherein training the mask-robust inpainting machine-learning model further comprises:

generating a relaxed mask that expands the initial mask; and

generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask,

wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

3 . The system of claim 2 , wherein the one or more processors further cause the system to:

generate the relaxed mask utilizing a mask refiner machine-learning model; and

train the mask refiner machine-learning model by tuning parameters of the mask refiner machine-learning model based on the measure of loss.

4 . The system of claim 1 , wherein training the mask-robust inpainting machine-learning model further comprises:

generating a perturbed mask from the initial mask;

generating a relaxed mask that expands the perturbed mask; and

generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask,

wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

5 . The system of claim 4 , wherein generating the perturbed mask comprises:

replacing the initial mask utilizing a free-form mask; or

eroding a selection of boundary pixels of the initial mask.

6 . The system of claim 4 , wherein generating the perturbed mask comprises:

identifying pixel regions within the initial mask;

determining probabilities that the pixel regions cover the object portrayed in the digital image; and

removing one or more of the pixel regions from the initial mask based on the probabilities.

7 . The system of claim 1 , wherein training the mask-robust inpainting machine-learning model further comprises:

generating, utilizing the mask-robust inpainting machine-learning model, an additional inpainted image from the digital image, wherein the additional inpainted image comprises additional modified pixels inside and outside the initial mask;

determining an additional measure of loss by comparing the additional inpainted image and the pseudo-ground-truth inpainted image; and

further tuning the parameters of the mask-robust inpainting machine-learning model based on the additional measure of loss.

8 . A computer-implemented method comprising training a mask-robust inpainting machine-learning model by:

generating, utilizing the mask-robust inpainting machine-learning model, an inpainted image from a digital image, wherein the inpainted image comprises modified pixels inside and outside an initial mask for an object portrayed in the digital image;

generating, utilizing an additional inpainting machine-learning model, a pseudo-ground-truth inpainted image from the digital image utilizing a dilated mask generated from the initial mask;

determining a measure of loss by comparing the inpainted image and the pseudo-ground-truth inpainted image; and

tuning parameters of the mask-robust inpainting machine-learning model based on the measure of loss.

9 . The computer-implemented method of claim 8 , wherein training the mask-robust inpainting machine-learning model further comprises:

generating a relaxed mask that expands the initial mask; and

generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask,

wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

10 . The computer-implemented method of claim 9 , further comprising:

generating the relaxed mask utilizing a mask refiner machine-learning model; and

training the mask refiner machine-learning model by tuning parameters of the mask refiner machine-learning model based on the measure of loss.

11 . The computer-implemented method of claim 8 , wherein training the mask-robust inpainting machine-learning model further comprises:

generating a perturbed mask from the initial mask;

generating a relaxed mask that expands the perturbed mask; and

generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask,

wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

12 . The computer-implemented method of claim 11 , wherein generating the perturbed mask comprises:

replacing the initial mask utilizing a free-form mask; or

eroding a selection of boundary pixels of the initial mask.

13 . The computer-implemented method of claim 11 , wherein generating the perturbed mask comprises:

identifying pixel regions within the initial mask;

determining probabilities that the pixel regions cover the object portrayed in the digital image; and

removing one or more of the pixel regions from the initial mask based on the probabilities.

14 . The computer-implemented method of claim 8 , wherein training the mask-robust inpainting machine-learning model further comprises:

generating, utilizing the mask-robust inpainting machine-learning model, an additional inpainted image from the digital image, wherein the additional inpainted image comprises additional modified pixels inside and outside the initial mask;

determining an additional measure of loss by comparing the additional inpainted image and the pseudo-ground-truth inpainted image; and

further tuning the parameters of the mask-robust inpainting machine-learning model based on the additional measure of loss.

15 . A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to train a mask-robust inpainting machine-learning model by:

generating, utilizing the mask-robust inpainting machine-learning model, an inpainted image from a digital image, wherein the inpainted image comprises modified pixels inside and outside an initial mask for an object portrayed in the digital image;

generating, utilizing an additional inpainting machine-learning model, a pseudo-ground-truth inpainted image from the digital image utilizing a dilated mask generated from the initial mask;

determining a measure of loss by comparing the inpainted image and the pseudo-ground-truth inpainted image; and

tuning parameters of the mask-robust inpainting machine-learning model based on the measure of loss.

16 . The non-transitory computer-readable medium of claim 15 , wherein training the mask-robust inpainting machine-learning model further comprises:

generating a relaxed mask that expands the initial mask; and

generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask,

wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

17 . The non-transitory computer-readable medium of claim 15 , wherein training the mask-robust inpainting machine-learning model further comprises:

generating a perturbed mask from the initial mask;

generating a relaxed mask that expands the perturbed mask; and

generating a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask,

wherein comparing the inpainted image and the pseudo-ground-truth inpainted image comprises comparing the modified image and the pseudo-ground-truth inpainted image.

18 . The non-transitory computer-readable medium of claim 17 , wherein generating the perturbed mask comprises:

replacing the initial mask utilizing a free-form mask; or

eroding a selection of boundary pixels of the initial mask.

19 . The non-transitory computer-readable medium of claim 17 , wherein generating the perturbed mask comprises:

identifying pixel regions within the initial mask;

determining probabilities that the pixel regions cover the object portrayed in the digital image; and

removing one or more of the pixel regions from the initial mask based on the probabilities.

20 . The non-transitory computer-readable medium of claim 15 , wherein training the mask-robust inpainting machine-learning model further comprises:

generating, utilizing the mask-robust inpainting machine-learning model, an additional inpainted image from the digital image, wherein the additional inpainted image comprises additional modified pixels inside and outside the initial mask;

determining an additional measure of loss by comparing the additional inpainted image and the pseudo-ground-truth inpainted image; and

further tuning the parameters of the mask-robust inpainting machine-learning model based on the additional measure of loss.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 26, 2023
From: AMIRGHODSI, SOHRAB; ZHANG, LINGZHI; BARNES, CONNELLY; SHECHTMAN, ELYA; ZHOU, YUQIAN; LIN, ZHE
To: ADOBE INC.
Reel/Frame 063452/0225 →
Continuity (1)
Related Publication 20240362757A1 · Oct 31, 2024
References Cited (49)
US 20220392025A1 · Mironica · 2022 [cited by examiner]
US 20230259587A1 · Lin · 2023 [cited by examiner]
US 20230342890A1 · Kanazawa · 2023 [cited by examiner]
US 20240046429A1 · Amirghodsi · 2024 [cited by examiner]
US 20240127412A1 · Lin · 2024 [cited by examiner]
Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE conference on computer vision and pattern recogni… [cited by applicant]
Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding. Image inpainting with learnable bidirectional attention maps In Proceedings of the IEEE/CVF International Conferenc… [cited by applicant]
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1438-1447, 2019. [cited by applicant]
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009. [cited by applicant]
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (EC… [cited by applicant]
Haitian Zheng, Zhe Lin, Jingwan Lu, Scott Cohen, Eli Shechtman, Connelly Barnes, Jianming Zhang, Ning Xu, Sohrab Amirghodsi, and Jiebo Luo. Cm-gan: Image inpainting with cascaded modulation gan and object-aware training… [cited by applicant]
Haoran Zhang, Zhenzhen Hu, Changzhi Luo, Wangmeng Zuo, and Meng Wang. Semantic image inpainting with progressive generative networks. In Proceedings of the 26th ACM international conference on Multimedia, pp. 1939-1947,… [cited by applicant]
Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4170-4179, 2019. [cited by applicant]
Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, and Jing Liao. Pd-gan: Probabilistic diverse gan for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.… [cited by applicant]
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4471-4480, 201… [cited by applicant]
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505… [cited by applicant]
Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse structure for image inpainting with hierarchical vq-vae. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1… [cited by applicant]
Jie Yang, Zhiquan Qi, and Yong Shi. Learning to incorporate structure knowledge for image inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12605-12612, 2020. [cited by applicant]
Jingyuan Li, Fengxiang He, Lefei Zhang, Bo Du, and Dacheng Tao. Progressive reconstruction of visual structure for image inpainting. In Proceedings of the IEEE/CVF Inter-national Conference on Computer Vision, pp. 5962-… [cited by applicant]
Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7760-7768, 2020. [cited by applicant]
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212, 2019. [cited by applicant]
Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang, Zhiwen 712 Zuo, Haibo Chen, Wei Xing, and Dongming Lu. Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In Proceedings of the IEEE/CVF Confe… [cited by applicant]
Liang Liao, Jing Xiao, Zheng Wang, Chia-Wen Lin, and Shinichi Satoh. Guidance and evaluation: Semantic-aware image inpainting for mixed scenes. In Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, Aug. 2… [cited by applicant]
Lingzhi Zhang, Connelly Barnes, Kevin Wampler, Sohrab Amirghodsi, Eli Shechtman, Zhe Lin, and Jianbo Shi. Inpainting at modern camera resolution by guided patchmatch with auto-curation. arXiv preprint arXiv:2208.03552, … [cited by applicant]
Lingzhi Zhang, Jiancong Wang, and Jianbo Shi. Multimodal image outpainting with regularized normalized diversification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3433-3442,… [cited by applicant]
Lingzhi Zhang, Yuqian Zhou, Connelly Barnes, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, and Jianbo Shi. Perceptual artifacts localization for inpainting. arXiv preprint arXiv:2208.03357, 2022. [cited by applicant]
Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. Advances in Neural Information Processing Systems, 33, 2020. [cited by applicant]
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417-424, 2000. [cited by applicant]
Ning Wang, Sihan Ma, Jingyuan Li, Yipeng Zhang, and Lefei Zhang. Multistage attention network for image inpainting. Pattern Recognition, 106:107448, 2020. [cited by applicant]
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Sü sstrunk. Slic superpixels. Technical report, 2010. [cited by applicant]
Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. Semantic image inpainting with deep generative models. In Proceedings of the IEEE conference on computer vision and pat… [cited by applicant]
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with … [cited by applicant]
Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG), 36(4):1-14, 2017. [cited by applicant]
Tengfei Wang, Hao Ouyang, and Qifeng Chen. Image inpainting with external-internal learning and monochromic bottleneck. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5120-5129… [cited by applicant]
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pp. 418-434, 2018. [cited by applicant]
Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Con-nelly Barnes, and Jiebo Luo. Foreground-aware image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5840-5848,… [cited by applicant]
Weiwei Cai and Zhanguo Wei. Piigan: generative adversarial networks for pluralistic image inpainting. IEEE Access, 8:48451-48463, 2020. [cited by applicant]
Xiefan Guo, Hongyu Yang, and Di Huang. Image inpainting via conditional texture and structure dual generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14134-14143, 2021. [cited by applicant]
Xintong Han, Zuxuan Wu, Weilin Huang, Matthew R Scott, and Larry S Davis. Finet: Compatible and diverse fashion image inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4481-4491… [cited by applicant]
Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-context encoder network for high-quality image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition… [cited by applicant]
Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Image inpainting via generative multi-column convolutional neural networks. arXiv preprint arXiv:1810.08771, 2018. [cited by applicant]
Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. Diverse image inpainting with bidirectional and autoregressive transformers. arXiv preprint ar… [cited by applicant]
Yu Zeng, Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, and Huchuan Lu. High-resolution image inpainting with iterative confidence feedback and guided upsampling. In European Conference on Computer Vision, pp. 1-17… [cited by applicant]
Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C-C Jay Kuo. Spg-net: Segmentation prediction and guidance network for image inpainting. arXiv preprint arXiv:1805.03356, 2018. [cited by applicant]
Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. Structureflow: Image inpainting via structure-aware appearance flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp… [cited by applicant]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conferenc… [cited by applicant]
Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, and Zhan Xu. Contextual residual aggregation for ultra high-resolution image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn… [cited by applicant]
Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao High-fidelity pluralistic image completion with transformers. arXiv preprint arXiv:2103.14031, 2021. [cited by applicant]
Zongyu Guo, Zhibo Chen, Tao Yu, Jiale Chen, and Sen Liu. Progressive image inpainting with full-resolution residual network. In Proceedings of the 27th acm international conference on multimedia, pp. 2496-2504, 2019. [cited by applicant]