IP Library › Granted Patent US 12,288,279
Granted Patent B2
US 12,288,279 · App. 18/058,538 · Granted Apr 29, 2025

Modifying digital images via scene-based editing using image understanding facilitated by artificial intelligence

Inventors: Jonathan Brandt (Santa Cruz, CA); Scott Cohen (Sunnyvale, CA); Zhe Lin (Fremont, CA); Zhihong Ding (Fremont, CA); Darshan Prasad (San Jose, CA); Matthew Joss (Seattle, WA); Celso Gomes (Bellevue, WA); Jianming Zhang (Campbell, CA); Olena Soroka (Hamburg, DE); Klaas Stoeckmann (Hamburg, DE); Michael Zimmermann (Hamburg, DE); Thomas Muehrke (Hamburg, DE)
Assignee: Adobe Inc.
G06T11/60G06F3/04842G06F3/04845G06T11/40
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,288,279
App. No.
18/058,538
Granted
Apr 29, 2025
Kind
B2
Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For instance, in one or more embodiments, the disclosed systems generate utilizing a segmentation neural network, an object mask for each object of a plurality of objects of a digital image. The disclosed systems detect a first user interaction with an object in the digital image displayed via a graphical user interface. The disclosed systems surface, via the graphical user interface, the object mask for the object in response to the first user interaction. The disclosed systems perform an object-aware modification of the digital image in response to a second user interaction with the object mask for the object.

Claims (58)

1. A system comprising:

at least one memory device; and

at least one processor configured to cause the system to:

receive a digital image portraying a plurality of objects;

pre-process the digital image in response to receiving the digital image and before receiving a user interaction with the digital image by:

generating, using a segmentation neural network, an object mask for each object of the plurality of objects; and

generating, using a hole filling model, a completed background for the digital image that includes generated background pixels behind each object;

detect user input to move or delete an object of the plurality of objects; and

modify the digital image in response to the user input by:

moving or deleting the object in accordance with the user input; and

exposing a set of background pixels behind the object from the completed background.

2. The system of claim 1 , wherein the at least one processor is configured to cause the system to generate, utilizing the hole filling model, the completed background for the digital image by generating, for at least one object of the digital image, a set of background pixels for behind the at least one object based on semantics of the digital image.

3. The system of claim 1 , wherein the at least one processor is further configured to cause the system to:

detect an additional user input for moving or deleting an additional object from the plurality of objects; and

modify the digital image by moving or deleting the additional object in accordance with the additional user input and exposing an additional set of background pixels behind the additional object from the completed background.

4. The system of claim 1 , wherein the at least one processor is configured to cause the system to:

detect the user input for moving or deleting the object from the plurality of objects by detecting a plurality of user input for deleting the plurality of objects of the digital image; and

modify the digital image by moving or deleting the object in accordance with the user input by modifying the digital image by deleting the plurality of objects to reveal the completed background generated using the hole filling model.

5. A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

pre-processing a digital image before receiving user input for modifying the digital image by:

generating, utilizing a segmentation neural network, an object mask for each object of a plurality of objects of the digital image;

generating a content fill for each object mask utilizing a hole filling model; and

generating a completed background for the digital image by placing each content fill behind the object mask for which each content fill was generated;

detecting, after generation of the completed background, a user-input interaction to move or delete an object in the digital image; and

exposing the content fill behind the object mask for the object upon moving or deleting the object.

6. The non-transitory computer-readable medium of claim 5 , wherein exposing the content fill behind the object mask upon moving or deleting the object comprises exposing a portion of the completed background associated with the content fill upon moving or deleting the object.

7. The non-transitory computer-readable medium of claim 5 , wherein generating the completed background comprises generating the completed background before detecting the user input to move or delete the object.

8. The non-transitory computer-readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

providing the digital image for display within a graphical user interface in response to receiving the digital image; and

maintaining, before detecting the user-input interaction to move or delete the object, the digital image for display within the graphical user interface, the digital image having the content fill for each object mask covered by a corresponding object.

9. The non-transitory computer-readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

providing the digital image for display within a graphical user interface;

detecting a user selection of the object via the graphical user interface; and

provide, via the graphical user interface in response to the user selection, a visual indication of the object mask for the object.

10. The non-transitory computer-readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

exposing the content fill behind the object mask for the object upon moving or deleting the object by exposing the content fill behind the object mask for the object upon moving the object from a first position to a second position within the digital image; and

maintaining the completed background for the digital image after moving the object by maintaining background pixels associated with the second position to which the object is moved.

11. The non-transitory computer-readable medium of claim 5 , wherein:

generating the content fill for the object mask of the object comprises completing at least part of a second object positioned behind the object; and

exposing the content fill behind the object mask comprises exposing the completed second object.

12. The non-transitory computer-readable medium of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

detecting an additional user input to move or delete the second object; and

exposing an additional content fill behind the second object upon moving or deleting the second object.

13. A computer-implemented method comprising:

pre-processing a digital image, before detecting a user interaction with the digital image, to enable object-based editing of the digital image by generating, utilizing a segmentation neural network, an object mask for each object of a plurality of objects of the digital image; and

after pre-processing the digital image:

detecting a first user interaction with an object in the digital image displayed via a graphical user interface;

surfacing, via the graphical user interface, the object mask for the object in response to the first user interaction; and

performing an object-aware modification of the digital image in response to a second user interaction with the object mask for the object.

14. The computer-implemented method of claim 13 , wherein surfacing, via the graphical user interface, the object mask for the object in response to the first user interaction comprises providing, for display within the graphical user interface, a visual indication of the object mask in association with the object.

15. The computer-implemented method of claim 13 , wherein performing the object-aware modification of the digital image comprises removing the object from an initial position within the digital image.

16. The computer-implemented method of claim 15 , further comprising:

pre-processing the digital image before detecting the user interaction with the digital image by generating a content fill for the object mask for the object; and

exposing, in response to removing the object from the initial position, the content fill for the object mask within the digital image.

17. The computer-implemented method of claim 16 , further comprising placing, before detecting the first user interaction with the object, the content fill behind the object mask at the initial position of the object within the digital image.

18. The computer-implemented method of claim 16 , wherein generating the content fill comprises generating the content fill utilizing an inpainting neural network.

19. The computer-implemented method of claim 13 , wherein pre-processing the digital image by generating the object mask for each object of the plurality of objects of the digital image before detecting the user interaction with the digital image comprises generating the object mask for each object upon receiving the digital image and before detecting the user interaction with the digital image.

20. The computer-implemented method of claim 13 , wherein performing the object-aware modification of the digital image in response to the second user interaction with the object mask for the object comprises deleting the object from the digital image or moving the object within the digital image using the object mask for the object.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 30, 2022
From: BRANDT, JONATHAN; COHEN, SCOTT; LIN, ZHE; DING, ZHIHONG; PRASAD, DARSHAN; JOSS, MATTHEW; GOMES, CELSO; ZHANG, JIANMING; SOROKA, OLENA; STOECKMANN, KLAAS; ZIMMERMANN, MICHAEL; MUEHRKE, THOMAS
To: ADOBE INC.
Reel/Frame 061928/0471 →
Continuity (1)
Related Publication 20240169624A1 · May 23, 2024
References Cited (215)
US 6204840B1 · Petelycky et al. · 2001 [cited by applicant]
US 6750890B1 · Sugimoto · 2004 [cited by applicant]
US 8125492B1 · Wainwright et al. · 2012 [cited by applicant]
US 9256621B2 · Lessin · 2016 [cited by applicant]
US 9405463B2 · Won et al. · 2016 [cited by applicant]
US 10192129B2 · Price et al. · 2019 [cited by applicant]
US 10249044B2 · Cheung et al. · 2019 [cited by applicant]
US 10460214B2 · Lu et al. · 2019 [cited by applicant]
US 10515160B1 · Jarvis · 2019 [cited by applicant]
US 10692221B2 · Pao et al. · 2020 [cited by applicant]
US 10748324B2 · Shechtman et al. · 2020 [cited by applicant]
US 10755447B2 · Chhaya et al. · 2020 [cited by applicant]
US 11055566B1 · Pham et al. · 2021 [cited by applicant]
US 11094083B2 · Eisenmann et al. · 2021 [cited by applicant]
US 11100917B2 · Bui et al. · 2021 [cited by applicant]
US 11107219B2 · Cohen et al. · 2021 [cited by applicant]
US 11126890B2 · Lin et al. · 2021 [cited by applicant]
US 11238362B2 · Jin et al. · 2022 [cited by applicant]
US 11373390B2 · Zhao et al. · 2022 [cited by applicant]
US 11462040B2 · Lin · 2022 [cited by examiner]
US 11538216B2 · Sunkavalli et al. · 2022 [cited by applicant]
US 11568627B2 · Price et al. · 2023 [cited by applicant]
US 11676390B2 · Tang et al. · 2023 [cited by applicant]
US 11682105B2 · Duan et al. · 2023 [cited by applicant]
US 12026845B2 · Pardeshi · 2024 [cited by applicant]
US 12175619B2 · Joachim · 2024 [cited by applicant]
US 12210800B2 · Soni et al. · 2025 [cited by applicant]
US 12223523B2 · Bhole et al. · 2025 [cited by applicant]
US 20080109717A1 · Krauter · 2008 [cited by applicant]
US 20090316957A1 · Chen et al. · 2009 [cited by applicant]
US 20130076773A1 · Chen et al. · 2013 [cited by applicant]
US 20130167087A1 · Tighe et al. · 2013 [cited by applicant]
US 20140081625A1 · Wilensky et al. · 2014 [cited by applicant]
US 20150356343A1 · Jain et al. · 2015 [cited by applicant]
US 20150365591A1 · Aronsson et al. · 2015 [cited by applicant]
US 20160180201A1 · Aubert et al. · 2016 [cited by applicant]
US 20170032551A1 · Fried et al. · 2017 [cited by applicant]
US 20170123641A1 · Lance · 2017 [cited by applicant]
US 20170347024A1 · Yanagi · 2017 [cited by applicant]
US 20170358092A1 · Bleibel et al. · 2017 [cited by applicant]
US 20190042871A1 · Pogorelik · 2019 [cited by applicant]
US 20190205643A1 · Liu et al. · 2019 [cited by applicant]
US 20190340798A1 · Garg · 2019 [cited by applicant]
US 20200042286A1 · Bui et al. · 2020 [cited by applicant]
US 20200051561A1 · Lai · 2020 [cited by applicant]
US 20200143838A1 · Peleg et al. · 2020 [cited by applicant]
US 20200151860A1 · Safdarnejad et al. · 2020 [cited by applicant]
US 20200160042A1 · Bui et al. · 2020 [cited by applicant]
US 20200302656A1 · Kumar et al. · 2020 [cited by applicant]
US 20200312298A1 · Bui et al. · 2020 [cited by applicant]
US 20210004576A1 · Bui et al. · 2021 [cited by applicant]
US 20210027448A1 · Cohen et al. · 2021 [cited by applicant]
US 20210027471A1 · Cohen et al. · 2021 [cited by applicant]
US 20210027497A1 · Ding et al. · 2021 [cited by applicant]
US 20210067344A1 · Drake et al. · 2021 [cited by applicant]
US 20210073267A1 · Chopra et al. · 2021 [cited by applicant]
US 20210103607A1 · Fisher · 2021 [cited by applicant]
US 20210142497A1 · Pugh et al. · 2021 [cited by applicant]
US 20210263962A1 · Chang et al. · 2021 [cited by applicant]
US 20210349674A1 · Britton · 2021 [cited by applicant]
US 20210374905A1 · Duan et al. · 2021 [cited by applicant]
US 20220058001A1 · Chang et al. · 2022 [cited by applicant]
US 20220068037A1 · Pardeshi · 2022 [cited by examiner]
US 20220129670A1 · Lin et al. · 2022 [cited by applicant]
US 20220129682A1 · Tang et al. · 2022 [cited by applicant]
US 20220237799A1 · Price et al. · 2022 [cited by applicant]
US 20220270215A1 · Lee · 2022 [cited by applicant]
US 20220284613A1 · Yin et al. · 2022 [cited by applicant]
US 20220337709A1 · Tsuruta · 2022 [cited by applicant]
US 20220383037A1 · Pham et al. · 2022 [cited by applicant]
US 20230103305A1 · Xu et al. · 2023 [cited by applicant]
US 20230127460A1 · Xie et al. · 2023 [cited by applicant]
US 20230147722A1 · Ramirez De Chanlatte et al. · 2023 [cited by applicant]
US 20230252752A1 · Cheng et al. · 2023 [cited by applicant]
US 20230326028A1 · Zhang et al. · 2023 [cited by applicant]
US 20240135511A1 · Singh et al. · 2024 [cited by applicant]
US 20240153047A1 · Smith · 2024 [cited by examiner]
US 20240169502A1 · Cohen et al. · 2024 [cited by applicant]
US 20240171848A1 · Figueroa et al. · 2024 [cited by applicant]
US 20240331322A1 · Smith · 2024 [cited by applicant]
US 20240355107A1 · Liba et al. · 2024 [cited by applicant]
US 20240412458A1 · Jampani et al. · 2024 [cited by applicant]
CA 3158287A1 · 2021 [cited by applicant]
CN 1512403A · 2004 [cited by applicant]
CN 110033475B · 2020 [cited by applicant]
GB 2623620A · 2024 [cited by applicant]
JP 2014235553A · 2014 [cited by applicant]
WO 2023250088A1 · 2023 [cited by applicant]
Chen, Hsiang-Ting et al. ACM Digital Library: Data-driven adaptive history for image editing, published Feb. 27, 2016, pp. 1-9. https://dl.acm.org/doi/10.1145/2856400.2856417 (Year: 2016). [cited by applicant]
Combined Search and Examination Report received in application No. GB2314585.7 dated Feb. 28, 2024. [cited by applicant]
Combined Search and Examination Report received in application No. GB2314323.3 dated Feb. 28, 2024. [cited by applicant]
Combined Search and Examination Report received in application No. GB2316991.5 dated May 8, 2024. [cited by applicant]
Combined Search and Examination Report received in application No. GB2317196.0 dated May 21, 2024. [cited by applicant]
Dhamo H, Farshad A, Laina I, Navab N, Hager GD, Tombari F, Rupprecht C. “Semantic Image Manipulation Using Scene Graphs”, 2020, IEEE/CVF Conf on Computer Vision and Pattern Recognition (CVPR), Jun. 13, 2020-Jun. 19, 202… [cited by applicant]
Draftback Replays a Google Document's Changes in Video Format!, Published Sep. 27, 2019, pp. 1-10, https://web.archive.org/web/20220902104215/https://terrystechtipscchs.blogspot.com/2019/09/draftback-replays-google-docu… [cited by applicant]
M. Choi, “Referring Object Manipulation of Natural Images with Conditional Classifier-free guidance”, Computer Vision ECCV 2022. ECCV 2022, Lecture Notes in Computer Science, vol. 13696. Springer Cham. https://doi.org/. [cited by applicant]
P.K. Saha et al., “Data Augmentation Technique to Expand Road Dataset Using Mask RCNN and Image Inpainting,” 2021 International Conference on Intelligent Technologies (CONIT), Hubli, India, 2021, pp. 1-6, doi: 10.1109/C… [cited by applicant]
U.S. Appl. No. 18/058,622, Mar. 21, 2024, Office Action. [cited by applicant]
U.S. Appl. No. 18/311,705, Mar. 27, 2024, Office Action. [cited by applicant]
Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, Chen Change Loy, “Self-Supervised Scene De-occlusion”, arXiv:2004.02788v1, Apr. 6, 2020, 11 pages. [cited by applicant]
U.S. Appl. No. 18/058,630, Nov. 15, 2023, Office Action. [cited by applicant]
Office Action as Received in CN Application No. 202311286113.8 dated Nov. 16, 2023. [cited by applicant]
U.S. Appl. No. 18/311,713, Jan. 4, 2024, Office Action. [cited by applicant]
B. AlBahar, J. Lu, J. Yang, Z. Shu, E. Shechtman, and J. Huang in Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN, SIGGRAPH Asia 2021 (Pose with Style). [cited by applicant]
G. Huang et al., Densely Connected Convolutional Networks, in IEEE Conference on Computer Vision and Pattern Recognition, 2016. [cited by applicant]
J. Ho, A. Jain, P Abbeel, Denoising Diffusion Probabilistic Models, arXiv:2006:11239. [cited by applicant]
Jaemin Cho et al., Fine-grained Image Captioning with CLIP Reward, arXiv:2205.13115, 2022. [cited by applicant]
Jiaming Song, et al. in Denoising diffusion implicit models. In ICLR, 2021. [cited by applicant]
O. Ronneberger, P. Fischer, and T. Brox, in U-net: Convolutional networks for biomedical image segmentation, MICCAI (3), vol. 9351 of Lecture Notes in Computer Science, p. 234-250 (2015). [cited by applicant]
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, in High-Resolution Image Synthesis with Latent Diffusion Models, arXiv:2112.10752v2. [cited by applicant]
Xihui Liu, Zhe Lin, Jianming Zhang, Handong Zhao, Quan Tran, Xiaogang Wang, and Hongsheng Li; “Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions”; arXiv:2008.01576v2[cs.CV] Apr. 21, 2021. [cited by applicant]
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, Abhinav Shrivastava; “Learning to Predict Visual Attributes in the Wild”; CVPR 2021. [cited by applicant]
Redmon, et al., You Only Look Once: Unified, Real-Time Object Detection, CVPR 2016. [cited by applicant]
Ning Xu et al., “Deep GrabCut for Object Selection,” published Jul. 14, 2017. [cited by applicant]
Lu Chi, Borui Jiang, and Yadong Mu in Fast Fourier convolution, Advances in Neural Information Processing Systems, 33 (2020). [cited by applicant]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila in Analyzing and improving the image quality of StyleGAN, Proc. CVPR (2020) (hereinafter StyleGan2). [cited by applicant]
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu in Semantic image synthesis with spatially-adaptive normalization, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019). [cited by applicant]
S. Ren, K. He, R. Girshick, and J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, NIPS, 2015. [cited by applicant]
Y. Li, W. Ouyang, B. Zhou, Y. Cui, J. Shi, and X. Wang, Factorizable net: An efficient subgraph based framework for scene graph generation, ECCV, Jun. 29, 2018. [cited by applicant]
F. Faghri et al., Vse++: Improving visual-semantic Embeddings with Hard Negatives, arXiv:1707.05612, 2017. [cited by applicant]
Liang-Chieh Chen et al., Rethinking Atrous Convolution for Semantic Image Segmentation, arXiv:1706.05587, 2017. [cited by applicant]
Liang-Chieh Chen et al., Deeplab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, arXiv:1606.00915, 2016. [cited by applicant]
Bin Ding, Chengjiang Long, Ling Zhang, Chunxia Xiao. “ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, … [cited by applicant]
Bin Liao, Yao Zhu, Chao Liang, Fei Luo, Chunxia Xiao: “Illumination animating and editing in a single picture using scene structure estimation.” Computers & Graphics (2019). [cited by applicant]
Bin Liu, Kun Xu, Ralph R. Martin: “Static scene illumination estimation from videos with applications.” Journal of Computer Science and Technology (2017). [cited by applicant]
Chao Tan, Xin Feng. “Unsupervised Shadow Removal Using Target Consistency Generative Adversarial Network.” arXiv preprint; arXiv:2010.01291v2, 14 pages, 2021. [cited by applicant]
Daquan Liu, Chengjiang Long, Hongpan Zhang, Hanning Yu, Xinzhi Dong, Chunxia Xiao: “ARshadowGAN: Shadow generative adversarial network for augmented reality in single light scenes.” CVPR (2020). In Proceedings of the IE… [cited by applicant]
Daquan Liu, Chengjiang Long, Hongpan Zhang, Hanning Yu, Xinzhi Dong, Chunxia Xiao. (2020). “ARShadowGAN.” [dataset]. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://github.com/ldq9526/ARSh… [cited by applicant]
Fangneng Zhan, Shijian Lu, Changgong Zhang, Feiying Ma, Xuansong Xie: “Adversarial Image Composition with Auxiliary Illumination.” ACCV 17 Pages. (2020). [cited by applicant]
Florin-Alexandru Vasluianu, Andres Romero, Luc Van Gool, and Radu Timofte. “Self-Supervised Shadow Removal.” arXiv preprint; arXiv:2010.11619v1. 10 pages, 2021. [cited by applicant]
Haitian Zheng, Zhe Lin, Jingwan Lu, Scott Cohen, Eli Shechtman, Connelly Barnes, Jianming Zhang, Ning Xu, Sohrab Amirghodsi, and Jiebo Luo. “Image Inpainting with Cascaded Modulation GAN and Object-Aware Training”. arXi… [cited by applicant]
Hieu Le and Dimitris Samaras. “Physics-based Shadow Image Decomposition for Shadow Removal.” IEEE Transactions on Pattern Analysis and Machine Intelligence. 14 Pages. 2021. [cited by applicant]
Hieu Le and Dimitris Samaras. “From Shadow Segmentation to Shadow Removal.” In proceedings of the European Conference on Computer Vision. 18 pages, 2020. [cited by applicant]
Hieu Le, Dimitris Samaras. “Shadow Removal via Shadow Image Decomposition.” International Conference on Computer Vision 2019. (ICCV). pp. 8578-8587. 2019. [cited by applicant]
Jiawei Liu; Qiang Wang; Huijie Fan; Jiandong Tian; Yandong Tang. “A Shadow Imaging Bilinear Model and Three-Branch Residual Network for Shadow Removal,” in IEEE Transactions on Neural Networks and Learning Systems, doi:… [cited by applicant]
Jiawei Liu; Qiang Wang; Huijie Fan; Wentao Li; Liangqiong Qu; Yandong Tang. “A Decoupled Multi-Task Network for Shadow Removal,” in IEEE Transactions on Multimedia, vol. 25, pp. 9449-9463, 2023, doi: 10.1109/TMM.2023.32… [cited by applicant]
Jifeng Wang, Xiang Li, Le Hui, Jian Yang. “Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal.” 2018 IEEE Conference on Computer Vision and Pattern Recognition (… [cited by applicant]
Jin Wan, Hui Yin, Zhenyao Wu, Xinyi Wu, Yanting Liu, and Song Wan. “Style-Guided Shadow Removal.” In proceedings of European Conference on Computer Vision 2022. 17 Pages. 2022. [cited by applicant]
Jin Wan, Hui Yin, Zhenyao Wu, Xinyi Wu, Zhihao Liu, Song Wang. “CRFormer: A Cross-Region Transformer for Shadow Removal.” arXiv preprint., arXiv:2207.01600v1. 11 Pages. 2022. [cited by applicant]
Kevin Karsch, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, Hailin Jin, Rafael Fonte, Michael Sittig, David Forsyth: “Automatic scene inference for 3d object compositing.” ACM Transactions on Graphics (2014). [cited by applicant]
Kevin Karsch, Varsha Hedau, David Forsyth, Derek Hoiem: “Rendering synthetic objects into legacy photographs.” ACM Transactions on Graphics (2011). [cited by applicant]
Kunpeng Niu; Yanli Liu; Enhua Wu; Guanyu Xing. “A Boundary-Aware Network for Shadow Removal.” IEEE Transactions on Multimedia (vol. 25), pp. 6782-6793, 2022. [cited by applicant]
Lan Fu, Changqing Zhou, Qing Guo, Felix Juefei-Xu, Hongkai Yu, Wei Feng, Yang Liu, Song Wang. “Auto-Exposure Fusion for Single-Image Shadow Removal.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern… [cited by applicant]
Lanqing Guo, Chong Wang, Wenhan Yang, Siyu Huang, Yufei Wang, Hanspeter Pfister, Bihan Wen. “ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal.” In CVPR 2023 pp. 14049-14058, 2023. [cited by applicant]
Lanqing Guo, Siyu Huang, Ding Liu, Hao Cheng, Bihan Wen. “ShadowFormer: Global Context Helps Image Shadow Removal.” In Proceedings AAAI Conference on Artificial Intelligence, 9 pages, 2023. [cited by applicant]
Lei Zhu, Zijun Deng, Xiaowei Hu, Chi-Wing Fu, Xuemiao Xu, Jing Qin, and Pheng-Ann Heng. “Bidirectional Feature Pyramid Network with Recurrent Attention Residual Modules for Shadow Detection.” Proceedings of the European… [cited by applicant]
Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, Liqing Zhang: “Making Images Real Again: A Comprehensive Survey on Deep Image Composition.” arXiv preprint arXiv:2106.14490 (2021). [cited by applicant]
Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, Rynson W. H. Lau. “DeshadowNet: A Multi-Context Embedding Deep Network for Shadow Removal.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR… [cited by applicant]
Ling Zhang, Chengjiang Long, Xiaolong Zhang, Chunxia Xiao. “RIS-GAN: Explore Residual and Illumination with Generative Adversarial Networks for Shadow Removal.” The Thirty-Fourth AAAI Conference on Artificial Intelligen… [cited by applicant]
Lucas Valença, Jinsong Zhang, Michaël Gharbi, Yannick Hold-Geoffroy, Jean-François Lalonde: “Shadow Harmonization for Realistic Compositing.” SIGGRAPH Asia (2023) 12 Pages. [cited by applicant]
Lvmin Zhang, Jinyue Jiang, Yi Ji, Chunping Liu. “SmartShadow: Artistic Shadow Drawing Tool for Line Drawings.” 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021, pp. 5371-5380,… [cited by applicant]
Mehmet Kerim Yucel, Valia Dimaridou, Bruno Manganelli, Mete Ozay, Anastasios Drosou, Albert Saa-Garriga. “LRA&LDRA: Rethinking Residual Predictions for Efficient Shadow Detection and Removal.” IEEE/CVF Winter Conference… [cited by applicant]
Mrinmoy Sen, Sai Pradyumna Chermala, Nazrinbanu Nurmohammad Nagori, Venkat Peddigari, Praful Mathur, B H Pawan Prasad, Moonhwan Jeong. “SHARDS: Efficient SHAdow Removal using Dual Stage Network for High-Resolution Image… [cited by applicant]
Naoto Inoue, Toshihiko Yamasaki. “Learning from Synthetic Shadows for Shadow Detection and Removal.” IEEE Transactions on Circuits and Systems for Video Technology (vol. 31, Issue: 11, Nov. 2021), pp. 4187-4197, 2020. [cited by applicant]
Qianhao Yu, Naishan Zheng, Jie Huang, and Feng Zhao. “CNSNet: A Cleanness-Navigated-Shadow Network for Shadow Removal.” European Conference on Computer Vision Workshop 2022. 18 pages. 2022. [cited by applicant]
Qingyang Liu, Jianting Wang, Li Niu. (2023). “Object Shadow Generation Dataset—DESOBAv2.” Version 2. [dataset]. AAAI 2023. https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBAv2. [cited by applicant]
Qingyuan Zheng, Zhuoru Li, Adam Bargteil. “Learning to Shadow Hand-Drawn Sketches”. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7436-7445, 2020. [cited by applicant]
Quanling Meng, Shengping Zhang, Zonglin Li, Chenyang Wang, Weigang Zhang, Qingming Huang: “Automatic Shadow Generation via Exposure Fusion.” IEEE Transactions on Multimedia, vol. 25, pp. 9044-9056, 2023, doi: 10.1109/TM… [cited by applicant]
Shengfeng He, Bing Peng; Junyu Dong; Yong Du, “Mask-ShadowNet: Toward Shadow Removal via Masked Adaptive Instance Normalization,” in IEEE Signal Processing Letters, vol. 28, pp. 957-961, 2021, doi: 10.1109/LSP.2021.3074… [cited by applicant]
Shuyang Zhang, Runze Liang, Miao Wang: “ShadowGAN: Shadow synthesis for virtual objects with conditional adversarial networks.” Computational Visual Media (2019). vol. 5, No. 1, Mar. 2019, 105-115. [cited by applicant]
Subhrajyoti Dasgupta, Arindam Das, Sudip Das, Andrei Bursuc, Ujjwal Bhattacharya and Senthil Yogamani. “UnShadowNet: Illumination Critic Guided Contrastive Learning For Shadow Removal.” arXiv preprint arXiv: 2203.15441v… [cited by applicant]
Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Yan Wang, Zejian Li, Lingyun Sun, Papa Mao, Ying Zang. “SAM Fails to Segment Anything?—SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical… [cited by applicant]
Tianyanshi Liu, Yuhang Li, Youdong Ding: “Shadow Generation for Composite Image with Multi-level Feature Fusion.” EITCE (2022). [cited by applicant]
Tianyu Wang, Xiaowei Hu, Chi-Wing Fu, and Pheng-Ann Heng. “Single-Stage Instance Shadow Detection with Bidirectional Relation Learning.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11 pag… [cited by applicant]
Tianyu Wang, Xiaowei Hu, Pheng-Ann Heng, Chi-Wing Fu. “Instance Shadow Detection with A Single-Stage Detector.” Institute of Electrical and Electronics Engineers (IEEE), IEEE Transactions on Pattern Analysis and Machine… [cited by applicant]
Tianyu Wang, Xiaowei Hu, Qiong Wang, Pheng-Ann Heng, and Chi-Wing Fu. “Instance Shadow Detection.” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1880-1889). 2021. [cited by applicant]
Xiaodong Cun, Chi-Man Pun, Cheng Shi. “Towards Ghost-Free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN.” Proceedings of the AAAI Conference on Artificial Intelligence, 34(07), 10680-10… [cited by applicant]
Xiaowei Hu, Chi-Wing Fu, Lei Zhu, Jing Qin, and Pheng-Ann Heng. “Direction-aware Spatial Context Features for Shadow Detection and Removal.” IEEE Transactions on Pattern Analysis and Machine Intelligence. 14 pages. 2019. [cited by applicant]
Xiaowei Hu, Lei Zhu, Chi-Wing Fu, Jing Qin, and Pheng-Ann Heng. “Direction-aware Spatial Context Features for Shadow Detection.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7454-7462, 2018. [cited by applicant]
Xiaowei Hu, Yitong Jiang, Chi-Wing Fu, Pheng-Ann Heng, “Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2472-2481, 2019. [cited by applicant]
Xinhao Tao, Junyan Cao, Yan Hong, Li Niu: “Shadow Generation with Decomposed Mask Prediction and Attentive Shadow Filling.” AAAI (2024). [cited by applicant]
Xinhao Tao, Junyan Cao, Yan Hong, Li Niu. “Rendered Shadow Generation Dataset—RDSOBA.” [dataset] AAAI 2024. https://github.com/bcmi/Rendered-Shadow-Generation-Dataset-RdSOBA. [cited by applicant]
Xuaner Cecilia Zhang, Jonathan T. Barron, Yun-Ta Tsai, Rohit Pandey, Xiuming Zhang, Ren Ng, and David E. Jacobs. “Portrait Shadow Manipulation.” In SIGGRAPH, CoRR abs/2005.08925, 14 pages, 2020. [cited by applicant]
Yan Hong, Li Niu, Jianfu Zhang: “Shadow Generation for Composite Image in Real-world Scenes.” AAAI arXiv:2104.10338v1 14 Pages. (2022). [cited by applicant]
Yan Hong, Li Niu, Jianfu Zhang. (2022). “Object Shadow Generation Dataset—DESOBA.” [dataset]. AAAI 2022. https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBA. [cited by applicant]
Yan Hong, Li Niu, Jianfu Zhang. “Shadow Generation for Composite Image in Real-World Scenes.” AAAI. (2022) . arXiv:2104.10338v3. 16 Pages. [cited by applicant]
Yaojie Liu, Andrew Hou, Xinyu Huang, Liu Ren, Xiaoming Liu. “Blind Removal of Facial Foreign Shadows.” In Proceedings of the British Machine Vision Conference 2022, 14 pages, 2022. [cited by applicant]
Yeying Jin, Aashish Sharma, and Robby T. Tan. “DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network.”. In Proceedings of the IEEE international conference on comput… [cited by applicant]
Yeying Jin, Wenhan Yang, Wei Ye, Yuan Yuan and Robby T. Tan. “ShadowDiffusion: Diffusion-based Shadow Removal using Classifier-driven Attention and Structure Preservation.” arXiv preprint arXiv:2211.08089v3. 10 pages. 2… [cited by applicant]
Yichen Sheng, Jianming Zhang, Bedrich Benes: “SSN: Soft shadow network for image compositing.” CVPR (2021). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4380-4390, 2021. [cited by applicant]
Yichen Sheng, Jianming Zhang, Julien Philip, Yannick Hold-Geoffroy, Xin Sun, He Zhang, Lu Ling, Bedrich Benes: “PixHt-Lab: Pixel Height Based Light Effect Generation for Image Compositing.” CVPR (2023). [cited by applicant]
Yichen Sheng, Yifan Liu, Jianming Zhang, Wei Yin, Oztireli Cengiz, He Zhang, Lin Zhe, Shechtman Eli, Bedrich Benes: “Controllable Shadow Generation Using Pixel Height Maps.” ECCV (2022). arXiv:2207.05385v2. Jul. 15, 202… [cited by applicant]
Yimin Xu, Mingbao Lin, Hong Yang, Fei Chao, Rongrong Ji. “Shadow-Aware Dynamic Convolution for Shadow Removal.” arXiv preprint. arXiv:2205.04908v3. 10 Pages. 2022. [cited by applicant]
Yingqing He, Yazhou Xing, Tianjia Zhang, and Qifeng Chen. 2021. “Unsupervised Portrait Shadow Removal via Generative Priors.” In Proceedings of the 29th ACM Int'l Conference on Multimedia (MM '21), Oct. 20-24, 2021, Vir… [cited by applicant]
Yurui Zhu, Jie Huang, Xueyang Fu, Feng Zhao, Qibin Sun, Zheng-Jun Zha. “Bijective Mapping Network for Shadow Removal.” 2022 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5627-5636. 2022. [cited by applicant]
Yurui Zhu, Xueyang Fu, Chengzhi Cao, Xi Wang, Qibin Sun, and Zheng-Jun Zha. 2022. “Single Image Shadow Detection via Complementary Mechanism.” In Proceedings of the 30th ACM International Conference on Multimedia (MM'22… [cited by applicant]
Yurui Zhu, Zeyu Xiao, Yanchi Fang, Xueyang Fu, Zhiwei Xiong, and Zheng-Jun Zha. (2022). Efficient Model-Driven Network for Shadow Removal. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3), 3635-3643.… [cited by applicant]
Zhenghao Xing, Tianyu Wang, Xiaowei Hu, Haoran Wu, Chi-Wing Fu, Pheng-Ann Heng. “Video Instance Shadow Detection.” arXiv preprint, arXiv:2211.12827v1, 10 pages, 2023. [cited by applicant]
Zhihao Liu, Hui Yin, Xinyi Wu, Zhenyao Wu, Yang Mi, Song Wang. “From Shadow Generation to Shadow Removal.” The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR). 10 pages, 2021. [cited by applicant]
Zhihao Liu; Hui Yin; Yang Mi; Mengyang Pu; Song Wang. “Shadow Removal by a Lightness-Guided Network With Training on Unpaired Data,” in IEEE Transactions on Image Processing, vol. 30, pp. 1853-1865, 2021, doi: 10.1109/T… [cited by applicant]
Zipei Chen, Chengjiang Long, Ling Zhang, Chunxia Xiao. “CANet: A Context-Aware Network for Shadow Removal.” International Conference on Computer Vision 2021 (ICCV2021). 13 Pages. 2021. [cited by applicant]
U.S. Appl. No. 18/058,630, Feb. 28, 2024, Notice of Allowance. [cited by applicant]
Combined Search and Examination Report received in application No. GB2317072.3 dated May 16, 2024. [cited by applicant]
Search Report received in GB2318128.2 dated Jun. 4, 2024. [cited by applicant]
Search Report received in GB2318199.3 dated Jun. 5, 2024. [cited by applicant]
Combined Search and Examination Report received in application No. GB2403106.4 dated Jul. 12, 2024. [cited by applicant]
U.S. Appl. No. 18/311,713, Jul. 5, 2024, Office Action. [cited by applicant]
U.S. Appl. No. 18/404,648, Jul. 17, 2024, Office Action. [cited by applicant]
Combined Search and Examination Report received in application GB 2403915.8 dated Aug. 9, 2024. [cited by applicant]
Search Report received in GB2402982.9 dated Jul. 10, 2024. [cited by applicant]
“10 things we love about Affinity Photo 1.7”, available on Jul. 25, 2021, pp. 1-6 https://web. archive.org/web/20210725082242/https://affinityspotlight. com/article/10-things-we-love-about-affi nity-photo-17 / (Year: 20… [cited by applicant]
U.S. Appl. No. 18/058,554, Dec. 6, 2024, Office Action. [cited by applicant]
U.S. Appl. No. 18/058,622, Oct. 8, 2024, Office Action. [cited by applicant]
U.S. Appl. No. 18/311,705, Dec. 6, 2024, Office Action. [cited by applicant]
U.S. Appl. No. 18/311,713, Nov. 21, 2024, Notice of Allowance. [cited by applicant]
U.S. Appl. No. 18/058,554, Aug. 6, 2024, Office Action. [cited by applicant]
U.S. Appl. No. 18/311,705, Sep. 13, 2024, Office Action. [cited by applicant]
Combined Search and Examination Report received in application No. GB2314323.3 dated Dec. 11, 2024. [cited by applicant]
Wang, Instance Shadow Detection (2020). [cited by applicant]
U.S. Appl. No. 18/404,648, Jan. 30, 2025, Notice of Allowance. [cited by applicant]
U.S. Appl. No. 18/058,575, Feb. 12, 2025, Office Action. [cited by applicant]
U.S. Appl. No. 18/058,554, Mar. 12, 2025, Notice of Allowance. [cited by applicant]
U.S. Appl. No. 18/058,622, Feb. 27, 2025, Office Action. [cited by applicant]
U.S. Appl. No. 18/320,664, Mar. 5, 2025, Office Action. [cited by applicant]
U.S. Appl. No. 18/460,365, Feb. 28, 2025, Notice of Allowance. [cited by applicant]
Cited By (4)
US 1,094,435 US 12,536,625 US 12,579,742 US 12,682,561