IP Library › Granted Patent US 12,682,426
Granted Patent B2
US 12,682,426 · App. 18/238,290 · Granted Jul 14, 2026

Artificial intelligence techniques for extrapolating HDR panoramas from LDR low FOV images

Inventors: Mohammad Reza Karimi Dastjerdi (Quebec, CA); Yannick Hold-Geoffroy (Quebec City, CA); Sai Bi (San Jose, CA); Jonathan Eisenmann (San Francisco, CA); Jean-François Lalonde (Quebec, CA)
Assignee: Adobe Inc.
G06T5/50G06T5/70G06T15/506G06T2200/24G06T2207/20081G06T2207/20084G06T2207/20092G06T2207/20208
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,682,426
App. No.
18/238,290
Filed
Aug 25, 2023
Granted
Jul 14, 2026
Kind
B2
Art Unit
2667
USPC
382/156
Abstract

In some examples, a computing system accesses a field of view (FOV) image that has a field of view less than 360 degrees and has low dynamic range (LDR) values. The computing system estimates lighting parameters from a scene depicted in the FOV image and generates a lighting image based on the lighting parameters. The computing system further generates lighting features generated the lighting image and image features generated from the FOV image. These features are aggregated into aggregated features and a machine learning model is applied to the image features and the aggregated features to generate a panorama image having high dynamic range (HDR) values.

Claims (37)

1 . A computer-implemented method in which one or more processing devices perform operations comprising:

accessing a field of view (FOV) image that has a field of view less than 360 degrees and has low dynamic range (LDR) values;

generating a lighting image based on lighting parameters estimated from a scene depicted in the FOV image;

generating, using a mapping module, a set of features from a random noise signal;

aggregating lighting features generated from the lighting image, image features generated from the FOV image, and the set of features generated from the random noise signal to generate aggregated features; and

applying a machine learning model to the image features and the aggregated features to generate a panorama image having high dynamic range (HDR) values.

2 . The computer-implemented method of claim 1 , further comprising updating the lighting parameters based on a user input, wherein the lighting image is generated based on updated lighting parameters.

3 . The computer-implemented method of claim 1 , wherein applying a machine learning model to the image features and the aggregated features to generate a panorama image having high dynamic range (HDR) values comprises:

applying a machine learning model to the image features and the aggregated features to generate a panorama image having low dynamic range (LDR) values; and

combining the lighting image with the panorama image having the LDR values to generate the high dynamic range (HDR) values.

4 . The computer-implemented method of claim 1 , wherein the lighting image is a spherical Gaussian image.

5 . The computer-implemented method of claim 1 , wherein the panorama image represents a field of view of 360 degrees.

6 . A system, comprising:

a memory component;

a processing device coupled to the memory component, the processing device to perform operations comprising:

generating a lighting image based on lighting parameters for a scene depicted in a field of view (FOV) image, the FOV image having a field of view less than 360 degrees and low dynamic range (LDR) values;

generating, using a mapping module, a set of features from a random noise signal;

aggregating the set of features, lighting features generated from the lighting image, and image features generated from the FOV image to generate aggregated features; and

generating a panorama image having high dynamic range (HDR) values based, at least in part, upon the image features and the aggregated features.

7 . The system of claim 6 , wherein the lighting parameters are specified by a user input or estimated from the scene and updated based on a user input.

8 . The system of claim 6 , wherein generate a panorama image having high dynamic range (HDR) values based, at least in part, upon the image features and the aggregated features comprises:

applying a machine learning model to the image features and the aggregated features to generate a panorama image having low dynamic range (LDR) values; and

combining the lighting image with the panorama image having the LDR values to generate the high dynamic range (HDR) values.

9 . The system of claim 8 , wherein the machine learning model is a generative adversarial network (GAN) model.

10 . The system of claim 6 , wherein the lighting image is a spherical Gaussian image.

11 . The system of claim 6 , wherein the panorama image represents a field of view of 360 degrees.

12 . A non-transitory computer-readable medium, storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

accessing a field of view (FOV) image that has a field of view less than 360 degrees and has low dynamic range (LDR) values;

generating a lighting image based on lighting parameters estimated from a scene depicted in the FOV image;

a step for generating aggregated features based on the lighting image; and

a step for applying a generative adversarial network (GAN) model to image features generated from the FOV image and the aggregated features to generate a panorama image having high dynamic range (HDR) values.

13 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise: updating the lighting parameters based on a user input, wherein the lighting image is generated based on updated lighting parameters.

14 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise: generating, using a mapping module, a set of features from a random noise signal, wherein the step for generating the aggregated features comprises aggregating lighting features generated from the lighting image, the image features, and the set of features.

15 . The non-transitory computer-readable medium of claim 12 , wherein the step for generating a panorama image having high dynamic range (HDR) values based on the image features and the aggregated features comprises:

combining the lighting image with the panorama image having the LDR values to generate the high dynamic range (HDR) values.

16 . The non-transitory computer-readable medium of claim 12 , wherein the lighting image is a spherical Gaussian image.

17 . The non-transitory computer-readable medium of claim 12 , wherein the panorama image represents a field of view of 360 degrees.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 25, 2023
From: DASTJERDI, MOHAMMAD REZA KARIMI; HOLD-GEOFFROY, YANNICK; BI, SAI; EISENMANN, JONATHAN; LALONDE, JEAN-FRANÇOIS
To: ADOBE INC.
Reel/Frame 064709/0207 →
Continuity (3)
Provisional Application 63456219 · Mar 31, 2023
Provisional Application 63416915 · Oct 17, 2022
Related Publication 20240127402A1 · Apr 18, 2024
References Cited (71)
US 10936909B2 · Sunkavalli et al. · 2021 [cited by applicant]
US 20180359416A1 · Hold-Geoffroy et al. · 2018 [cited by applicant]
US 20210073955A1 · Zhang et al. · 2021 [cited by applicant]
WO 2021042208A1 · 2021 [cited by applicant]
Gardner, M., Sunkavalli, K., Yumer, E., Shen, X., Gambaretto, E., Gagné, C., & Lalonde, J. (2017). Learning to predict indoor illumination from a single image. ACM Transactions on Graphics (TOG), 36, 1-14 (Year: 2017). [cited by examiner]
Zhu et al., Spatially-Varying Outdoor Lighting Estimation from Intrinsics, Vision and Pattern Recognition (CVPR), Apr. 2021, pp. 12834-12842 (Year: 2021). [cited by examiner]
GB Application No. GB2314638.4 Search and Examination Report, Mar. 4, 2024, 5 pages. [cited by applicant]
Akimoto et al., 360-Degree Image Completion by Two-Stage Conditional Gans, IEEE International Conference on Image Processing (ICIP), Sep. 22-25, 2019, 4 pages. [cited by applicant]
Akimoto et al., Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 18-24, 2022, pp. 11441-11450. [cited by applicant]
Barnes et al., PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing, ACM Transactions on Graphics, vol. 28, No. 3, Jul. 27, 2009, 10 pages. [cited by applicant]
Barron et al., Shape, Illumination, and Reflectance from Shading, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, No. 8, Aug. 2015, 19 pages. [cited by applicant]
Cheng et al., Learning Scene Illumination by Pairwise Photos from Rear and Front Mobile Cameras, Computer Graphics Forum, vol. 37, No. 7, Oct. 2018, 9 pages. [cited by applicant]
Dastjerdi et al., Guided Co-Modulated GAN for 360° Field of View Extrapolation, International Conference on 3D Vision (3DV), Available Online at: https://arxiv.org/pdf/2204.07286.pdf, Aug. 23, 2022, 11 pages. [cited by applicant]
Debevec, Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-Based Graphics with Global Illumination and High Dynamic Range Photography, SIGGRAPH '98: Proceedings of the 25th Annual Conference o… [cited by applicant]
Efros et al., Image Quilting for Texture Synthesis and Transfer, SIGGRAPH '01: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Aug. 2001, 6 pages. [cited by applicant]
Efros et al., Texture Synthesis By Non-Parametric Sampling, Proceedings of the Seventh IEEE International Conference on Computer Vision, Sep. 20-27, 1999, 6 pages. [cited by applicant]
Eilertsen et al., HDR Image Reconstruction from a Single Exposure Using Deep CNNs, ACM Transactions on Graphics, vol. 36, No. 6, Nov. 20, 2017, 15 pages. [cited by applicant]
Endo et al., Deep Reverse Tone Mapping, ACM Transactions on Graphics, vol. 36, No. 6, Nov. 20, 2017, pp. 177:1-177:10. [cited by applicant]
Gardner et al., Deep Parametric Indoor Lighting Estimation, Available Online at: https://openaccess.thecvf.com/content_ICCV_2019/papers/Gardner_Deep_Parametric_Indoor_Lighting_Estimation_ICCV_2019_paper.pdf, Oct. 2019, … [cited by applicant]
Gardner et al., Learning to Predict Indoor Illumination from a Single Image, ACM Transactions on Graphics, vol. 36, No. 6, Nov. 21, 2017, 14 pages. [cited by applicant]
Garon et al., Fast Spatially-Varying Indoor Lighting Estimation, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2019, pp. 6908-6917. [cited by applicant]
Griffiths et al., OutCast: Outdoor Single-Image Relighting with Cast Shadows, Computer Graphics Forum, vol. 41, No. 2, Apr. 20, 2022, 20 pages. [cited by applicant]
Guillemot et al., Image Inpainting: Overview and Recent Advances, IEEE Signal Processing Magazine, vol. 31, No. 1, Jan. 2014, pp. 127-144. [cited by applicant]
Hara et al., Spherical Image Generation from a Single Image by Considering Scene Symmetry, Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21), vol. 35, No. 2, May 18, 2021, pp. 1513-1521. [cited by applicant]
Hartley et al., Multiple View Geometry in Computer Vision, Cambridge University Press, ISBN:0521540518, Second Edition, 2004, 673 pages. [cited by applicant]
Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, Proceedings of the 31st International Conference on Neural Information Processing Systems, Available Online at: https://d… [cited by applicant]
Hold-Geoffroy et al., Deep Outdoor Illumination Estimation, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 7312-7321. [cited by applicant]
Hold-Geoffroy et al., Deep Sky Modeling for Single Image Outdoor Lighting Estimation, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2019, pp. 6927-6935. [cited by applicant]
Isola et al., Image-to-Image Translation with Conditional Adversarial Networks, Available Online at: https://arxiv.org/abs/1611.07004, Nov. 22, 2017, 17 pages. [cited by applicant]
Karras et al., A Style-Based Generator Architecture for Generative Adversarial Networks, Available Online at: https://arxiv.org/pdf/1812.04948.pdf, Mar. 29, 2019, pp. 1-12. [cited by applicant]
Kulkarni et al., 360FusionNeRF: Panoramic Neural Radiance Fields with Joint Guidance, Available Online at: https://arxiv.org/pdf/2209.14265.pdf, Sep. 28, 2022, pp. 1-8. [cited by applicant]
Lalonde et al., Estimating the Natural Illumination Conditions from a Single Outdoor Image, International Journal of Computer Vision, vol. 98, No. 2, Jun. 1, 2012, pp. 123-145. [cited by applicant]
Lalondew et al., Lighting Estimation in Outdoor Image Collections, 2014 2nd International Conference on 3D Vision, Dec. 8-11, 2014, 8 pages. [cited by applicant]
Lalonde et al., The Laval HDR Sky Database, Available at http://www.hdrdb.com/, 2016, 2 pages. [cited by applicant]
Lalonde et al., What Do the Sun and the Sky Tell US About the Camera?, International Journal of Computer Vision, vol. 88, No. 1, May 2010, pp. 24-51. [cited by applicant]
Legendre et al., DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality, Computer Vision and Pattern Recognition, Available online at: https://arxiv.org/pdf/1904.01175.pdf, Apr. 2, 2019, 11 pages. [cited by applicant]
Li et al., Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-varying Lighting and SVBRDF from a Single Image, Computer Vision and Pattern Recognition, Available online at: https://arxiv.org/pdf/1905.02722.pd… [cited by applicant]
Li et al., Physically-based Editing of Indoor Scene Lighting from a Single Image, Computer Vision and Pattern Recognition, Available online at: https://arxiv.org/pdf/2205.09343.pdf, Jul. 23, 2022, 35 pages. [cited by applicant]
Liu et al., Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline, Image and Video Processing, Available online at: https://arxiv.org/pdf/2004.01179.pdf, Apr. 2, 2020, 10 pages. [cited by applicant]
Mandl et al., Learning Lightprobes for Mixed Reality Illumination, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Available online at: https://www.cs.ubc.ca/~kmyi/pubs/2017/mandl2017learning/p… [cited by applicant]
Marnerides et al., ExpandNet: A Deep Convolutional Neural Network for High Dynamic Range Expansion from Low Dynamic Range Content, Computer Vision and Pattern Recognition, vol. 37, No. 2, Available online at: https://ar… [cited by applicant]
Mildenhall et al., NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, Computer Vision and Pattern Recognition, Available online at: https://arxiv.org/pdf/2003.08934.pdf, Aug. 3, 2020, 25 pages. [cited by applicant]
Paszke et al., PyTorch: An Imperative Style, High-Performance Deep Learning Library, Advances in Neural Information Processing Systems 32, Dec. 3, 2019, pp. 8026-8037. [cited by applicant]
Radford et al., Learning Transferable Visual Models from Natural Language Supervision, Computer Vision and Pattern Recognition, Available online at: https://arxiv.org/pdf/2103.00020.pdf, Feb. 26, 2021, 42 pages. [cited by applicant]
Radford et al., Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Under review as a conference paper at ICLR 2016, arXiv:1511.06434v2, Jan. 7, 2016, pp. 1-16. [cited by applicant]
Ramamoorthi et al., An Efficient Representation for Irradiance Environment Maps, SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, Available online at: https://csew… [cited by applicant]
Sengupta et al., Neural Inverse Rendering of an Indoor Scene from a Single Image, Computer Vision and Pattern Recognition, Available online at: https://arxiv.org/pdf/1901.02453.pdf, Sep. 14, 2019, 21 pages. [cited by applicant]
Shu et al., Neural Face Editing with Intrinsic Image Disentangling, IEEE Conference on Computer Vision and Pattern Recognition (CVPR'17), Available online at: https://arxiv.org/pdf/1704.04131v1.pdf, Apr. 13, 2017, 22 pa… [cited by applicant]
Somanath et al., HDR Environment Map Estimation for Real-Time Augmented Reality, Computer Vision and Pattern Recognition, Available online at: https://arxiv.org/pdf/2011.10687.pdf, Jul. 27, 2021, 16 pages. [cited by applicant]
Song et al., Neural Illumination: Lighting Prediction for Indoor Environments, Computer Vision and Pattern Recognition, Available online at: https://arxiv.org/pdf/1906.07370.pdf, Jun. 18, 2019, 17 pages. [cited by applicant]
Srinivasan et al., Lighthouse: Predicting Lighting Volumes for Spatially-coherent Illumination, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Available online at: https://arxiv.org/pdf/2003… [cited by applicant]
Tang et al., Estimating Spatially-varying Lighting in Urban Scenes with Disentangled Representation, European Conference on Computer Vision, 2022, pp. 1-16. [cited by applicant]
Tewari et al., State of the Art on Neural Rendering, Computer Graphics Forum, vol. 39, No. 2, Available Online at: https://arxiv.org/pdf/2004.03805.pdf, Apr. 8, 2020, 27 pages. [cited by applicant]
Wang et al., High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Aug. 20, 2018, pp. 8798-8807. [cited by applicant]
Wang et al., Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting, Available Online at: https://arxiv.org/pdf/2109.06061.pdf, Oct. 20, 2021, 10 pages. [cited by applicant]
Wang et al., Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion, Available Online at: https://arxiv.org/pdf/2208.09480.pdf, Aug. 19, 2022, pp. 1-17. [cited by applicant]
Wang et al., StyleLight: HDR Panorama Generation for Lighting Estimation and Editing, Available Online at: https://arxiv.org/pdf/2207.14811.pdf, Jul. 29, 2022, pp. 1-22. [cited by applicant]
Weber et al., Editable Indoor Lighting Estimation, Available Online at: https://arxiv.org/pdf/2211.03928.pdf, Nov. 2022, pp. 1-16. [cited by applicant]
Wiles et al., SynSin: End-to-end View Synthesis from a Single Image, Available Online at: https://arxiv.org/pdf/1912.08804.pdf, Apr. 18, 2020, 22 pages. [cited by applicant]
Yu et al., Luminance Attentive Networks for HDR Image and Panorama Reconstruction, Computer Graphics Forum, vol. 40, No. 7, Sep. 14, 2021, pp. 181-192. [cited by applicant]
Zhan et al., EMLight: Lighting Estimation via Spherical Distribution Approximation, The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21), Dec. 21, 2020, pp. 3287-3295. [cited by applicant]
Zhan et al., GMLight: Lighting Estimation via Geometric Distribution Approximation, IEEE Transactions on Image Processing, vol. 31, Feb. 20, 2021, pp. 1-12. [cited by applicant]
Zhan et al., Sparse Needlets for Lighting Estimation with Spherical Transport Loss, Available Online at: https://arxiv.org/pdf/2106.13090.pdf, Jun. 24, 2021, 11 pages. [cited by applicant]
Zhang et al., All-Weather Deep Outdoor Lighting Estimation, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2019, pp. 10158-10166. [cited by applicant]
Zhang et al., Fixup Initialization: Residual Learning Without Normalization, Available Online at: https://arxiv.org/pdf/1901.09321.pdf, Mar. 12, 2019, pp. 1-16. [cited by applicant]
Zhang et al., Learning High Dynamic Range from Outdoor Panoramas, IEEE International Conference on Computer Vision (ICCV), Mar. 2017, pp. 4519-4528. [cited by applicant]
Zhao et al., Large Scale Image Completion via Co-Modulated Generative Adversarial Networks, Available Online at: https://arxiv.org/pdf/2103.10428.pdf, Mar. 18, 2021, pp. 1-25. [cited by applicant]
Zhao et al., PointAR: Efficient Lighting Estimation for Mobile Augmented Reality, Available Online at: https://arxiv.org/pdf/2004.00006.pdf, Jul. 17, 2020, pp. 1-16. [cited by applicant]
Zheng et al., Semantic Layout Manipulation with High-Resolution Sparse Attention, Available Online at: https://arxiv.org/pdf/2012.07288.pdf, Dec. 14, 2020, 17 pages. [cited by applicant]
Zhu et al., IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes, Available Online at: https://arxiv.org/pdf/2206.08423.pdf, Jun. 16, 2022, 19 pages. [cited by applicant]
Zhu et al., Spatially-Varying Outdoor Lighting Estimation from Intrinsics, Vision and Pattern Recognition (CVPR), Apr. 2021, pp. 12834-12842. [cited by applicant]