IP Library › Granted Patent US 12,373,920
Granted Patent B2
US 12,373,920 · App. 17/658,770 · Granted Jul 29, 2025

Recommending objects for image composition using a geometry-and-lighting aware neural network

Inventors: Zhe Lin (Fremont, CA); Sijie Zhu (Orlando, FL); Jason Wen Yong Kuen (Santa Clara, CA); Scott Cohen (Sunnyvale, CA); Zhifei Zhang (San Jose, CA)
Assignee: Adobe Inc.
G06T5/50G06T3/60G06T5/70G06T7/194G06T2207/20084G06T2207/20221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,373,920
App. No.
17/658,770
Granted
Jul 29, 2025
Kind
B2
Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media that utilizes artificial intelligence to learn to recommend foreground object images for use in generating composite images based on geometry and/or lighting features. For instance, in one or more embodiments, the disclosed systems transform a foreground object image corresponding to a background image using at least one of a geometry transformation or a lighting transformation. The disclosed systems further generating predicted embeddings for the background image, the foreground object image, and the transformed foreground object image within a geometry-lighting-sensitive embedding space utilizing a geometry-lighting-aware neural network. Using a loss determined from the predicted embeddings, the disclosed systems update parameters of the geometry-lighting-aware neural network. The disclosed systems further provide a variety of efficient user interfaces for generating composite digital images.

Claims (73)

1. A computer-implemented method comprising:

generating a transformed foreground object image from a foreground object image corresponding to a background image by modifying content of the foreground object image using at least one of a geometry transformation or a lighting transformation; and

optimizing parameters of a geometry-lighting-aware neural network to decrease a distance between positive samples and increase a distance between negative samples and to enable retrieval of foreground objects that are compatible with background images by:

generating, utilizing a geometry-lighting-aware neural network, predicted embeddings for the background image, the foreground object image, and the transformed foreground object image within a geometry-lighting-sensitive embedding space by:

generating, utilizing a background network of the geometry-lighting-aware neural network, a first predicted embedding for the background image; and

generating, utilizing a foreground network of the geometry-lighting-aware neural network, a second predicted embedding for the foreground object image and a third predicted embedding for the transformed foreground object image; and

updating parameters of the geometry-lighting-aware neural network utilizing a loss determined from the predicted embeddings.

2. The computer-implemented method of claim 1 , wherein modifying the content of the foreground object image using at least one of the geometry transformation or the lighting transformation comprises modifying the content of the foreground object image using the lighting transformation by:

modifying a digital image using a Gaussian blur;

extracting a portion of the modified digital image that corresponds to the foreground object image using a segmentation mask corresponding to the foreground object image; and

modifying the content of the foreground object image using the extracted portion of the modified digital image.

3. The computer-implemented method of claim 1 , wherein modifying the content of the foreground object image using at least one of the geometry transformation or the lighting transformation comprises modifying the content of the foreground object image using the geometry transformation by:

modifying the foreground object image using one or more homography transformations; and

flipping the modified foreground object image using a flipping transformation.

4. The computer-implemented method of claim 1 , further comprising:

identifying an additional foreground object image as a negative sample with respect to the background image; and

generating, utilizing the geometry-lighting-aware neural network, an additional predicted embedding for the additional foreground object image within the geometry-lighting-sensitive embedding space,

wherein updating the parameters of the geometry-lighting-aware neural network utilizing the loss determined from the predicted embeddings comprises updating the parameters of the geometry-lighting-aware neural network utilizing the loss determined from the predicted embeddings and the additional predicted embedding.

5. The computer-implemented method of claim 4 , further comprising determining the loss from the predicted embeddings and the additional predicted embedding by:

determining a first triplet loss using a first set of predicted embeddings for the background image, the foreground object image, and the additional foreground object image;

determining a second triplet loss using a second set of predicted embeddings for the background image, the foreground object image, and the transformed foreground object image; and

combining the first triplet loss and the second triplet loss.

6. The computer-implemented method of claim 1 , further comprising:

generating the foreground object image by extracting a foreground object from a digital image utilizing a segmentation mask; and

generating the background image by modifying the digital image to cover a portrayal of the foreground object within the digital image with a mask.

7. The computer-implemented method of claim 1 ,

wherein updating the parameters of the geometry-lighting-aware neural network utilizing the loss determined from the predicted embeddings comprises updating parameters of the background network of the geometry-lighting-aware neural network utilizing the loss while maintaining parameters of the foreground network of the geometry-lighting-aware neural network; and

further comprising updating the parameters of the foreground network utilizing an additional loss determined from additional predicted embeddings while maintaining the parameters of the background network.

8. The computer-implemented method of claim 1 , further comprising utilizing the geometry-lighting-aware neural network with the updated parameters to recommend at least one foreground object image for use with at least one background image in generating a composite image.

9. The computer-implemented method of claim 1 , wherein:

generating the first predicted embedding for the background image using the background network comprises generating, using the background network, a first embedding that encodes a first set of lighting features or geometry features of the background image;

generating the second predicted embedding for the foreground object image using the foreground network comprises generating, using the foreground network, a second embedding that encodes a second set of lighting features or geometry features of the foreground object image; and

generating the third predicted embedding for the transformed foreground object image using the foreground network comprises generating, using the foreground network, a third embedding that encodes a third set of lighting features or geometry features of the transformed foreground object image, the third set of lighting features or geometry features included one or more features created via the geometry transformation or the lighting transformation.

10. A system comprising:

one or more memory devices; and

one or more processors configured to cause the system to optimize parameters for a geometry-lighting-aware neural network to decrease a distance between positive samples and increase a distance between negative samples via an alternating learning process by:

generating, utilizing the geometry-lighting-aware neural network, predicted embeddings for a foreground object image and a background image corresponding to the foreground object image within a geometry-lighting-sensitive embedding space by:

generating, utilizing a background network of the geometry-lighting-aware neural network, a first predicted embedding that encodes a first set of lighting features or geometry features of the background image; and

generating, utilizing a foreground network of the geometry-lighting-aware neural network, a second predicted embedding that encodes a second set of lighting features or geometry features of the foreground object image;

updating parameters of the background network utilizing the predicted embeddings while maintaining parameters of the foreground network;

generating, utilizing the geometry-lighting-aware neural network, additional predicted embeddings for at least one foreground object image and at least one background image corresponding to the at least one foreground object image within the geometry-lighting-sensitive embedding space; and

updating the parameters of the foreground network utilizing the additional predicted embeddings while maintaining the parameters of the background network.

11. The system of claim 10 , wherein the one or more processors are further configured to cause the system to:

generate, from the foreground object image, a transformed foreground object image by modifying content of the foreground object image utilizing a geometry transformation or a lighting transformation;

generate, utilizing the geometry-lighting-aware neural network, a predicted embedding for the transformed foreground object image within the geometry-lighting-sensitive embedding space; and

update the parameters of the background network utilizing the predicted embedding for the transformed foreground object image and the predicted embeddings for the foreground object image and the background image.

12. The system of claim 11 , wherein the one or more processors are configured to cause the system to generate the transformed foreground object image from the foreground object image utilizing the lighting transformation by modifying the content of the foreground object image utilizing a Gaussian blur.

13. The system of claim 10 , wherein the one or more processors are further configured to cause the system to:

determine a first set of triplet losses from the predicted embeddings utilizing a first triplet loss function and a second triplet loss function; and

determine a second set of triplet losses from the additional predicted embeddings utilizing the first triplet loss function and the second triplet loss function.

14. The system of claim 13 , wherein:

updating the parameters of the background network utilizing the predicted embeddings comprises updating the parameters of the background network utilizing the first set of triplet losses; and

updating the parameters of the foreground network utilizing the additional predicted embeddings comprises updating the parameters of the foreground network utilizing the second set of triplet losses.

15. The system of claim 10 , wherein the one or more processors are further configured to cause the system to:

generate the foreground object image by extracting a foreground object from a digital image utilizing an eroded segmentation mask corresponding to the foreground object; and

generate the background image by modifying the digital image to cover a portrayal of the foreground object within the digital image utilizing an extended mask.

16. A non-transitory computer-readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:

generating a transformed foreground object image from a foreground object image corresponding to a background image by modifying content of the foreground object image using at least one of a geometry transformation or a lighting transformation; and

optimizing parameters of a geometry-lighting-aware neural network to decrease a distance between positive samples and increase a distance between negative samples and to enable retrieval of foreground objects that are compatible with background images by:

generating, utilizing a geometry-lighting-aware neural network, predicted embeddings for the background image, the foreground object image, and the transformed foreground object image within a geometry-lighting-sensitive embedding space by:

generating, utilizing a background network of the geometry-lighting-aware neural network, a first predicted embedding for the background image; and

generating, utilizing a foreground network of the geometry-lighting-aware neural network, a second predicted embedding for the foreground object image and a third predicted embedding for the transformed foreground object image; and

updating parameters of the geometry-lighting-aware neural network utilizing a loss determined from the predicted embeddings.

17. The non-transitory computer-readable medium of claim 16 , wherein:

generating the first predicted embedding for the background image using the background network comprises generating the first predicted embedding using a first convolutional neural network; and

generating the second predicted embedding for the foreground object image using the foreground network comprises generating the second predicted embedding using a second convolutional neural network.

18. The non-transitory computer-readable medium of claim 16 , wherein updating the parameters of the geometry-lighting-aware neural network utilizing the loss determined from the predicted embeddings comprises updating a first set of parameters of the background network utilizing the loss while maintaining a second set of parameters of the foreground network.

19. The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise:

generating, utilizing the geometry-lighting-aware neural network, additional predicted embeddings for an additional background image, an additional foreground object image, and an additional transformed foreground object image within the geometry-lighting-sensitive embedding space; and

updating the second set of parameters of the foreground network utilizing an additional loss determined from the additional predicted embeddings while maintaining the updated first set of parameters of the background network.

20. The non-transitory computer-readable medium of claim 19 , wherein:

updating the first set of parameters of the background network utilizing the loss comprises updating the first set of parameters of the background network using one or more triplet loss functions; and

updating the second set of parameters of the foreground network utilizing the additional loss comprises updating the second set of parameters of the foreground network using the one or more triplet loss functions.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 11, 2022
From: LIN, ZHE; ZHU, SIJIE; KUEN, JASON WEN YONG; COHEN, SCOTT; ZHANG, ZHIFEI
To: ADOBE INC.
Reel/Frame 059564/0048 →
Continuity (1)
Related Publication 20230325991A1 · Oct 12, 2023
References Cited (49)
US 5579471A · Barber et al. · 1996 [cited by applicant]
US 10504264B1 · Koenig · 2019 [cited by applicant]
US 20080301546A1 · Moore et al. · 2008 [cited by applicant]
US 20140035950A1 · Jonsson · 2014 [cited by applicant]
US 20190251401A1 · Shechtman · 2019 [cited by examiner]
US 20190361994A1 · Shen et al. · 2019 [cited by applicant]
US 20200033615A1 · Kim · 2020 [cited by examiner]
US 20200134858A1 · Yang · 2020 [cited by examiner]
US 20220019849A1 · Kim et al. · 2022 [cited by applicant]
US 20230138380A1 · Chen · 2023 [cited by examiner]
DE 1020002964A1 · 2021 [cited by applicant]
GB 2574087A · 2019 [cited by applicant]
WO 2020102767A1 · 2020 [cited by applicant]
Pixabay; Stunning free images & royalty free stock; Date downloaded May 26, 2022; https://pixabay.com. [cited by applicant]
Samaneh Azadi, Deepak Pathak, Sayna Ebrahimi, and Trevor Darrell. Compositional gan: Learning image-conditional binary composition. International Journal of Computer Vision, 128(10):2570-2585, 2020. [cited by applicant]
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11(3), 2010. [cited by applicant]
Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Manivasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun. Geosim: Realistic video simulation via geometry-aware composition for self-driving.… [cited by applicant]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009. [cited by applicant]
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010. [cited by applicant]
Priya Goyal, Piotr Dollar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large mini-batch sgd: Training imagenet in 1 hour. arXiv preprint arXiv… [cited by applicant]
Yifan Jiang, He Zhang, Jianming Zhang, Yilin Wang, Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, and Zhangyang Wang. Ssh: A self-supervised framework for image harmonization. In Proceedings of t… [cited by applicant]
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [cited by applicant]
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of … [cited by applicant]
Jean-Francois Lalonde, Derek Hoiem, Alexei A Efros, Carsten Rother, John Winn, and Antonio Criminisi. Photo clip art. ACM transactions on graphics (TOG), 26(3):3-es, 2007. [cited by applicant]
Donghoon Lee, Sifei Liu, Jinwei Gu, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Context-aware synthesis and placement of object instances. arXiv preprint arXiv:1812.02350, 2018. [cited by applicant]
Youngwan Lee and Jongyoul Park. Centermask: Real-time anchor-free instance segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13906-13915, 2020. [cited by applicant]
Xueting Li, Sifei Liu, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, and Jan Kautz. Putting humans in a scene: Learning affordance in 3d indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision an… [cited by applicant]
Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey. St-gan: Spatial transformer generative adversarial networks for image compositing. In Proceedings of the IEEE Conference on Computer Vision and P… [cited by applicant]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-7… [cited by applicant]
Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, and Liqing Zhang. Making images real again: A comprehensive survey on deep image composition.arXiv preprint arXiv:2106.14490, 2021. [cited by applicant]
Rohit Pandey, Sergio Orts Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. Total relighting: learning to relight portraits for background replacement. ACM Tra… [cited by applicant]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning libra… [cited by applicant]
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556, 2014. [cited by applicant]
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9627-9636, 2019. [cited by applicant]
Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-Hsuan Yang. Deep image harmonization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3789-3797, 2017. [cited by applicant]
Yifan Wang, Andrew Liu, Richard Tucker, Jiajun Wu, Brian L Curless, Steven M Seitz, and Noah Snavely. Repopulating street scenes. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. … [cited by applicant]
Zongze Wu, Dani Lischinski, and Eli Shechtman. Fine-grained foreground retrieval via teacher-student learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3646-3654, 2021. [cited by applicant]
He Zhang, Jianming Zhang, Federico Perazzi, Zhe Lin, and Vishal M Patel. Deep image compositing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 365-374, 2021. [cited by applicant]
Lingzhi Zhang, Tarmily Wen, Jie Min, Jiancong Wang, David Han, and Jianbo Shi. Learning object placement by in-painting for compositional data augmentation. In Computer Vision—ECCV 2020: 16th European Conference, Glasgo… [cited by applicant]
Hengshuang Zhao, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Brian Price, and Jiaya Jia. Compositing-aware image search. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 502-516, 2018. [cited by applicant]
Yinan Zhao, Brian Price, Scott Cohen, and Danna Gurari. Unconstrained foreground object search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2030-2039, 2019. [cited by applicant]
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641… [cited by applicant]
Jun-Yan Zhu, Philipp Krahenbuhl, Eli Shechtman, and Alexei A Efros. Learning a discriminative model for the perception of realism in composite images. In Proceedings of the IEEE International Conference on Computer Visi… [cited by applicant]
U.S. Appl. No. 17/658,774, filed Nov. 27, 2024, Office Action. [cited by applicant]
M. Eitz, R. Richter, K. Hildebrand, T. Boubekeur and M. Alexa, “Photosketcher: Interactive Sketch-Based Image Synthesis,” in IEEE Computer Graphics and Applications, vol. 31, No. 6, pp. 56-66, Nov.-Dec. 2011, doi: 10.11… [cited by applicant]
U.S. Appl. No. 17/658,774, filed Mar. 17, 2025, Office Action. [cited by applicant]
U.S. Appl. No. 18/167,690, filed Feb. 24, 2025, Office Action. [cited by applicant]
Neural Filters. Adobe Photoshop, 2020 [retrieved on Dec. 5, 2020]. Retrieved from the Internet Archive Wayback Machine. [cited by applicant]
U.S. Appl. No. 18/167,690, Jun. 20, 2025, Office Action. [cited by applicant]