IP Library › Granted Patent US 12,530,839
Granted Patent B2
US 12,530,839 · App. 18/505,009 · Granted Jan 20, 2026

Relightable neural radiance field model

Inventors: Derek Edward Bradley (Zürich, CH); Prashanth Chandran (Zürich, CH); Paulo Fabiano Urnau Gotardo (Zürich, CH); Yingyan Xu (Zürich, CH); Gaspard Zoss (Zürich, CH)
Assignees: Disney Enterprises, INC.; ETH Zürich (Eidgenössische Technische Hochschule Zürich)
G06T15/506G06T7/90G06T19/20G06T2207/10024G06T2207/20081G06T2207/20084G06T2210/21G06T2219/2012
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,530,839
App. No.
18/505,009
Granted
Jan 20, 2026
Kind
B2
Abstract

The present invention sets forth a technique for generating two-dimensional (2D) renderings of a three-dimensional (3D) scene from an arbitrary camera position under arbitrary lighting conditions. This technique includes determining, based on a plurality of 2D representations of a 3D scene, a radiance field function for a neural radiance field (NeRF) model. This technique further includes determining, based on a plurality of 2D representations of a 3D scene, a radiance field function for a “one light at a time” (OLAT) model. The technique further includes rendering a 2D representation of the scene based on a given camera position and illumination data. The technique further includes computing a rendering loss based on the difference between the rendered 2D representation and an associated one of the plurality of 2D representations of the scene. The technique further includes modifying at least one of the NeRF and OLAT models based on the rendering loss.

Claims (47)

1 . A computer-implemented method for performing scene rendering, the computer-implemented method comprising:

determining, for each of a plurality of three-dimensional (3D) locations in a 3D scene via execution of a first trained machine learning model, a density value associated with the 3D location;

determining, for each of the plurality of 3D locations via execution of a second trained machine learning model, a diffuse color value and a specular color value associated with the 3D location based on a given camera location and a given lighting map;

determining a pixel color value for each of a plurality of pixels in a two-dimensional (2D) representation of the scene based on the density values, the diffuse color values, and the specular color values associated with the plurality of 3D locations; and

generating, based on the pixel color values associated with the plurality of pixels, a 2D rendering of the scene.

2 . The computer-implemented method of claim 1 , further comprising:

generating, using the first trained machine learning model, a neural feature vector based on the 3D location; and

transmitting the neural feature vector to the second trained machine learning model.

3 . The computer-implemented method of claim 1 , wherein the first trained machine learning model is a neural radiance field (NeRF) model and the second trained machine learning model is a “one light at a time” (OLAT) model.

4 . The computer-implemented method of claim 1 , wherein the lighting map includes position information associated with each of a plurality of light sources.

5 . The computer-implemented method of claim 4 , further comprising:

generating, based on the 3D location and the lighting map, a light direction that represents a direction from one of the plurality of light sources to the 3D location in the scene;

calculating, using a spherical codebook, an OLAT code based on the light direction; and

transmitting the OLAT code to the second trained machine learning model,

wherein the second trained machine learning model is a “one light at a time” (OLAT) model.

6 . The computer-implemented method of claim 5 , wherein calculating the OLAT code further comprises multiplying a spherical harmonic parameterization of the light direction by a matrix of learned coefficients included in the spherical codebook.

7 . The computer-implemented method of claim 5 , wherein for each 3D location, the steps of generating the light direction, calculating the OLAT code based on the light direction, and transmitting the OLAT code to the second trained machine learning model are repeated for each of the plurality of light sources included in the lighting map.

8 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

determining, for each of a plurality of three-dimensional (3D) locations in a 3D scene via execution of a first trained machine learning model, a density value associated with the 3D location;

determining, for each of the plurality of 3D locations via execution of a second trained machine learning model, a diffuse color value and a specular color value associated with the 3D location based on a given camera location and a given lighting map;

determining a pixel color value for each of a plurality of pixels in a two-dimensional (2D) representation of the scene based on the density values, diffuse color values, and specular color values associated with the plurality of 3D locations; and

generating, based on the pixel color values associated with the plurality of pixels, a 2D rendering of the scene.

9 . The one or more non-transitory computer-readable media of claim 8 , wherein the instructions further cause the one or more processors to perform the steps of:

generating, using the first trained machine learning model, a neural feature vector based on the 3D location; and

transmitting the neural feature vector to the second trained machine learning model.

10 . The one or more non-transitory computer-readable media of claim 8 , wherein the first trained machine learning model is a neural radiance field (NeRF) model and the second trained machine learning model is a one light at a time (OLAT) model.

11 . The one or more non-transitory computer-readable media of claim 8 , wherein the lighting map includes position information associated with each of a plurality of light sources.

12 . The one or more non-transitory computer-readable media of claim 11 , wherein the instructions further cause the one or more processors to perform the steps of:

generating, based on the 3D location and the lighting map, a light direction that represents a direction from one of the plurality of light sources included in the lighting map to the 3D location in the scene;

calculating, using a spherical codebook, an OLAT code based on the light direction; and

transmitting the OLAT code to the second trained machine learning model.

13 . The one or more non-transitory computer-readable media of claim 12 , wherein calculating the OLAT code further comprises multiplying a spherical harmonic parameterization the light direction by a matrix of learned coefficients included in the spherical codebook.

14 . The one or more non-transitory computer-readable media of claim 12 , wherein for each 3D location, the steps of generating the light direction, calculating the OLAT code based on the light direction, and transmitting the OLAT code to the second trained machine learning model are repeated for each of the plurality of light sources included in the lighting map.

15 . A computer-implemented method for performing scene rendering, the computer-implemented method comprising:

determining, based on a plurality of two-dimensional (2D) representations of a three-dimensional (3D) scene, a first radiance field function associated with a first machine learning model;

determining, based on the plurality of 2D representations of the 3D scene, a second radiance field function associated with a second machine learning model;

generating a combined radiance field function based on the radiance field functions associated with the first machine learning model and the second machine learning model;

generating, based on the combined radiance field function, a color value for a pixel in a 2D rendering of the scene;

computing a rendering loss based on a difference between the color value for the pixel and a ground truth color value associated with a corresponding pixel in a corresponding one of the plurality of 2D representations; and

modifying at least one of the first machine learning model or the second machine learning model based on the rendering loss.

16 . The computer-implemented method of claim 15 , further comprising:

generating, using the first machine learning model, a neural feature vector based on a 3D location in the 3D scene; and

transmitting the neural feature vector to the second machine learning model.

17 . The computer-implemented method of claim 15 , wherein the first machine learning model is a neural radiance field (NeRF) model and the second machine learning model is a “one light at a time” (OLAT) model.

18 . The computer-implemented method of claim 15 , wherein the rendering loss is a mean-squared error measurement of the difference between the color value for the pixel and the ground truth color value.

19 . The computer-implemented method of claim 15 , further comprising iteratively training the first machine learning model and the second machine learning model until the rendering loss is below a predetermined threshold.

20 . The computer-implemented method of claim 15 , wherein each of the plurality of 2D representations of the 3D scene includes positioning information for a camera and for one or more light sources.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 11, 2024
From: THE WALT DISNEY COMPANY (SWITZERLAND) GMBH
To: DISNEY ENTERPRISES, INC.
Reel/Frame 066104/0098 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 10, 2024
From: BRADLEY, DEREK EDWARD; CHANDRAN, PRASHANTH; URNAU GOTARDO, PAULO FABIANO; XU, YINGYAN; ZOSS, GASPARD
To: THE WALT DISNEY COMPANY (SWITZERLAND) GMBH; ETH ZÜRICH (EIDGENÖSSISCHE TECHNISCHE HOCHSCHULE ZÜRICH)
Reel/Frame 066080/0402 →
Continuity (2)
Provisional Application 63383459 · Nov 11, 2022
Related Publication 20240161391A1 · May 16, 2024
References Cited (61)
Ben Mildenhall, NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, Aug. 3, 2020, Google Research (Year: 2020). [cited by examiner]
Tiancheng Sun, Light Stage Super-Resolution: Continuous High-Frequency Relighting, Dec. 2020, ACM Trans. Graph., vol. 39, No. 6, Article 261 (Year: 2020). [cited by examiner]
Eisko, “Animatable Digital Double of Louise by Eisko”, Retrieved from https://eisko.com/louise/, on Nov. 10, 2023, pp. 1-10. [cited by applicant]
Barron et al., “Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields”, CVPR, 2022, pp. 5470-5479. [cited by applicant]
Beeler et al., “High-Quality Single-Shot Capture of Facial Geometry”, ACM Transactions on Graphics, http://doi.acm.org/10.1145/1778765.1778777, vol. 29, No. 4, Article 40, Jul. 2010, pp. 40:1-40:9. [cited by applicant]
Bi et al., “Deep Relightable Appearance Models for Animatable Faces”, ACM Transactions on Graphics, https://doi.org/10.1145/3450626.3459829, vol. 40, No. 4, Article 89, Aug. 2021, pp. 89:1-89:15. [cited by applicant]
Bi et al., “Neural Reflectance Fields for Appearance Acquisition”, arXiv:2008.03824, Aug. 16, 2020, pp. 1-11. [cited by applicant]
Bi et al., “Deep Reflectance vols. Relightable Reconstructions from Multi-View Photometric Images”, arXiv:2007.09892, Jul. 20, 2020, pp. 1-21. [cited by applicant]
Boss et al., “NeRD: Neural Reflectance Decomposition from Image Collections”, IEEE/CVF International Conference on Computer Vision, 2021, pp. 12684-12694. [cited by applicant]
Boss et al., “SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections”, arXiv:2205.15768, May 31, 2022, pp. 1-20. [cited by applicant]
Boss et al., “Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition”, 35th Conference on Neural Information Processing Systems, 2021, pp. 1-14. [cited by applicant]
Brox et al., “High Accuracy Optical Flow Estimation Based on a Theory for Warping”, Springer, ECCV, vol. 3024, 2004, 13 pages. [cited by applicant]
Chan et al., “Efficient Geometry-aware 3D Generative Adversarial Networks”, arXiv:2112.07945, Apr. 27, 2022, 27 pages. [cited by applicant]
Chen et al., “TensoRF: Tensorial Radiance Fields”, In Computer Vision-ECCV 2022: 17th European Conference, 2022, pp. 1-18. [cited by applicant]
Debevec et al., “Acquiring the Reflectance Field of a Human Face”, SIGGRAPH, 2000, pp. 1-12. [cited by applicant]
Fridovich-Keil et al., “Plenoxels: Radiance Fields without Neural Networks”, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5501-5510. [cited by applicant]
Gafni et al., “Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction”, CVPR, 2021, pp. 8649-8658. [cited by applicant]
Garbin et al., “FastNeRF: High-Fidelity Neural Rendering at 200FPS”, ICCV, 2021, pp. 14346-14355. [cited by applicant]
Guo et al., “The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting”, ACM Transactions on Graphics, https://doi.org/10.1145/3355089.3356571, vol. 38, No. 6, Article 217, Nov. 2019, pp. 217:… [cited by applicant]
Hong et al., “HeadNeRF: A Real-time NeRF-based Parametric Head Model”, CVPR, 2022, pp. 20374-20384. [cited by applicant]
Kuang et al., “NeROIC: Neural Rendering of Objects from Online Image Collections”, ACM Transactions on Graphics, https://doi.org/10.1145/3528223.3530177, vol. 41, No. 4, Article 56, Jul. 2022, pp. 56:1-56:19. [cited by applicant]
Li et al., “EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes”, ACM Transactions on Graphics, https://doi.org/10.1145/3528223.3530130, vol. 41, No. 4, Article 166, Jul… [cited by applicant]
Liu et al., “Learning Smooth Neural Functions via Lipschitz Regularization”, SIGGRAPH, Conference Proceedings, https://doi.org/10.1145/3528233.3530713, Aug. 7-11, 2022, pp. 1-13. [cited by applicant]
Lyu et al., “Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination”, ECCV, 2022, pp. 1-18. [cited by applicant]
Ma et al., “Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination”, Eurographics Symposium on Rendering, The Eurographics Association, 2007, 12 pages. [cited by applicant]
Martin-Brualla et al., “NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections”, CVPR, 2021, pp. 7210-7219. [cited by applicant]
Meka et al., “Deep Reflectance Fields- High-Quality Facial Reflectance Field Inference from Color Gradient Illumination”, ACM Transactions on Graphics, https://doi.org/10.1145/3306346.3323027, vol. 38, No. 4, Article 77… [cited by applicant]
Meka et al., “Deep Relightable Textures—Volumetric Performance Capture with Neural Rendering”, ACM Transactions on Graphics, https://doi.org/10.1145/3414685.3417814, vol. 39, No. 6, Article 259, Dec. 2020, pp. 259:1-259… [cited by applicant]
Mildenhall et al., “NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images”, arXiv:2111.13679, Nov. 26, 2021, pp. 1-18. [cited by applicant]
Mildenhall et al., “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, arXiv:2003.08934, Aug. 3, 2020, pp. 1-25. [cited by applicant]
Müller et al., “Instant Neural Graphics Primitives with a Multiresolution Hash Encoding”, ACM Transactions on Graphics, https://doi.org/10.1145/3528223.3530127, vol. 41, No. 4, Article 102, Jul. 2022, pp. 102:1-102:15. [cited by applicant]
Munkberg et al., “Extracting Triangular 3D Models, Materials, and Lighting From Images”, CVPR, 2022, pp. 8280-8290. [cited by applicant]
Nestmeyer et al., “Learning Physics-guided Face Relighting under Directional Light”, CVPR, 2020, pp. 5124-5133. [cited by applicant]
Niemeyer et al., “GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields”, CVPR, 2021, pp. 11453-11464. [cited by applicant]
Nimeroff et al., “Efficient Re-rendering of Naturally Illuminated Environments”, 5th Eurographics Workshop on Rendering, Jun. 1994, pp. 1-15. [cited by applicant]
Oechsle et al., “UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction”, ICCV, 2021, pp. 5589-5599. [cited by applicant]
Pandey et al., “Total Relighting: Learning to Relight Portraits for Background Replacement”, ACM Transactions on Graphics, https://doi.org/10.1145/3450626.3459872, vol. 40, No. 4, Article 1, Aug. 2021, pp. 1:1-1:21. [cited by applicant]
Park et al., “Nerfies: Deformable Neural Radiance Fields”, ICCV, 2021, pp. 5865-5874. [cited by applicant]
Park et al., “HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields”, arXiv:2106.13228, Article 1, Sep. 10, 2021, pp. 1:1-1:16. [cited by applicant]
Pumarola et al., “D-NeRF: Neural Radiance Fields for Dynamic Scenes”, arXiv:2011.13961, Nov. 27, 2020, pp. 1-10. [cited by applicant]
Riviere et al., “Single-Shot High-Quality Facial Geometry and Skin Appearance Capture”, ACM Transactions on Graphics, https://doi.org/10.1145/3386569.3392464, vol. 39, No. 4, Article 81, Jul. 2020, pp. 81:1-81:12. [cited by applicant]
Rudnev et al., “Neural Radiance Fields for Outdoor Scene Relighting”, arXiv:2112.05140, Dec. 9, 2021, pp. 1-13. [cited by applicant]
Srinivasan et al., “NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis”, CVPR, 2021, pp. 7495-7504. [cited by applicant]
Sun et al., “NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting”, Eurographics Symposium on Rendering, arXiv:2107.12351, Jul. 26, 2021, 12 pages. [cited by applicant]
Tancik et al., “Block-NeRF: Scalable Large Scene Neural View Synthesis”, CVPR, 2022, pp. 8248-8258. [cited by applicant]
Turki et al., “Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs”, CVPR, 2022, pp. 12922-12931. [cited by applicant]
Verbin et al., “Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields”, In 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5491-5500. [cited by applicant]
Wang et al., “MoRF: Morphable Radiance Fields for Multiview Neural Head Modeling”, SIGGRAPH Conference Proceedings, https://doi.org/10.1145/3528233.3530753, Aug. 7-11, 2022, 9 pages. [cited by applicant]
Wang et al., “NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction”, 35th Conference on Neural Information Processing Systems, 2021, pp. 1-13. [cited by applicant]
Xu et al., “Improved Lighting Models for Facial Appearance Capture”, The Eurographics Association, vol. 41, No. 2, 2022, 4 pages. [cited by applicant]
Yang et al., “PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo”, arXiv:2207.11406, Dec. 22, 2022, pp. 1-28. [cited by applicant]
Yao et al., “NelLF: Neural Incident Light Field for Physically-based Material Estimation”, 2022, 22 pages. [cited by applicant]
Yu et al., “PlenOctrees for Real-time Rendering of Neural Radiance Fields”, ICCV, 2021, pp. 5752-5761. [cited by applicant]
Zhang et al., “NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild”, 35th Conference on Neural Information Processing Systems, 2021, pp. 1-13. [cited by applicant]
Zhang et al., “IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images”, CVPR, 2022, pp. 5565-5574. [cited by applicant]
Zhang et al., “Neural Light Transport for Relighting and View Synthesis”, ACM Transactions on Graphics, https://doi.org/10.1145/3446328, vol. 40, No. 1, Jan. 20, 2021, pp. 1-16. [cited by applicant]
Zhang et al., “NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination”, ACM Transactions on Graphics, https://doi.org/10.1145/3478513.3480496, vol. 40, No. 6, Article 237, Dec. 2021, pp. … [cited by applicant]
Zhang et al., “Modeling Indirect Illumination for Inverse Rendering”, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18643-18652. [cited by applicant]
Zheng et al., “Neural Relightable Participating Media Rendering”, 35th Conference on Neural Information Processing Systems, 2021, pp. 1-13. [cited by applicant]
Xu et al., “ReNeRF: Relightable Neural Radiance Fields with Nearfield Lighting (Supplementary Material)”, 2023, 7 pages. [cited by applicant]
Xu et al., “ReNeRF: Relightable Neural Radiance Fields with Nearfield Lighting”, ICCV, 2023, pp. 22581-22591. [cited by applicant]