IP Library › Granted Patent US 12,499,673
Granted Patent B2
US 12,499,673 · App. 18/074,371 · Granted Dec 16, 2025

Large scene neural view synthesis

Inventors: Vincent Michael Casser (Cambridge, MA); Henrik Kretzschmar (Mountain View, CA); Matthew Justin Tancik (Albuquerque, NM); Sabeek Mani Pradhan (San Francisco, CA); Benjamin Joseph Mildenhall (London, GB); Pratul Preeti Srinivasan (San Francisco, CA); Jonathan Tilton Barron (Alameda, CA)
Assignee: Waymo LLC
G06V10/82G06T5/50G06V10/761G06T2207/10024G06T2207/20221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,499,673
App. No.
18/074,371
Granted
Dec 16, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for rendering a new image that depicts a scene from a perspective of a camera at a new camera viewpoint.

Claims (63)

1 . A method performed by one or more computers, the method comprising:

maintaining a plurality of view synthesis models, wherein each view synthesis model is configured to receive an input specifying a camera viewpoint and to generate as output a synthesized image of a scene from the camera viewpoint;

obtaining an input specifying a new camera viewpoint;

selecting, based on the new camera viewpoint, a subset of the plurality of view synthesis models;

for each view synthesis model in the subset, processing a respective input specifying the new camera viewpoint to generate as output a synthesized image of the scene from the new camera viewpoint; and

combining the synthesized images generated by the view synthesis models in the subset to generate a final synthesized image of the scene from the new camera viewpoint, wherein the combining comprises:

determining a respective weight for each view synthesis model in the subset; and

generating the final synthesized image by interpolating between the synthesized images generated by the view synthesis models in the subset in accordance with the respective weights for the view synthesis models in the subset.

2 . The method of claim 1 , wherein each view synthesis model has been trained on a respective set of training data that includes images captured from viewpoints within a corresponding sub-region of the scene.

3 . The method of claim 2 , wherein the view synthesis models have been trained independently on the respective sets of training data.

4 . The method of claim 1 , wherein selecting, based on the new camera viewpoint, a subset of the plurality of view synthesis models comprises:

selecting, for inclusion in the subset, each view synthesis model that corresponds to a respective sub-region of the scene that includes the new camera viewpoint.

5 . The method of claim 4 , further comprising:

determining, for each view synthesis model that corresponds to a respective sub-region of the scene that includes the new camera viewpoint, a respective visibility estimate that estimates a degree to which points along rays cast from the new viewpoint were visible in training images used to train the view synthesis model; and

removing, from the subset, any view synthesis model that has a respective visibility estimate that is below a visibility threshold.

6 . The method of claim 4 , wherein determining the respective weight for each view synthesis model in the subset comprises, for each view synthesis model:

determining the respective weight based on a distance between the new camera viewpoint and a center of the corresponding sub-region of the scene.

7 . The method of claim 5 , wherein each view synthesis model comprises:

a first neural network that is configured to receive a first input comprising data representing coordinates of a point in the scene and process the first input to generate an output comprising a volume density for the point and a feature vector; and

a second neural network that is configured to receive a second input comprising the feature vector and data representing a viewing direction and process the second input to generate as output a color.

8 . The method of claim 7 , wherein processing a respective input specifying the new camera viewpoint to generate as output a synthesized image of the scene from the new camera viewpoint comprises, for each pixel in the image:

sampling a plurality of points along a ray from the new camera viewpoint and along a viewing direction that corresponds to the pixel;

for each sampled point:

generating a first input comprising data representing coordinates of the sampled point;

processing the first input using the first neural network in the view synthesis model to generate an output comprising a volume density for the sampled point and a feature vector;

generating a second input comprising the feature vector and data representing the viewing direction corresponding to the pixel; and

processing the second input using the second neural network in the view synthesis model to generate as output a color for the sampled point; and

generating a color for the pixel using the colors and volume densities for the sampled points.

9 . The method of claim 8 , wherein, for each view synthesis model, the second input comprises a respective appearance embedding characterizing a target appearance of the synthesized image.

10 . The method of claim 9 , further comprising:

receiving a target appearance embedding for a first view synthesis model in the subset; and

generating the respective appearance embeddings for the other view synthesis models in the subset based on the target appearance embedding for the first view synthesis model.

11 . The method of claim 9 , further comprising:

receiving a target appearance embedding; and

setting the respective appearance embeddings for the view synthesis models in the subset to the target appearance embedding.

12 . The method of claim 8 wherein for each view synthesis model, the second input comprises data representing target camera exposure information for the synthesized image.

13 . The method of claim 7 , wherein each view synthesis model comprises a third neural network that is configured to receive a third input comprising data representing coordinates of the point in the scene and data representing the viewing direction and to process the third input to output an estimated transmittance of the point from the viewing direction, and wherein determining, for each view synthesis model that corresponds to a respective sub-region of the scene that includes the new viewpoint, a respective visibility estimate comprises:

for each of a plurality of point—viewing direction pairs, processing a third input comprising data representing coordinates of the point in the pair and data representing the viewing direction in the pair using the third neural network in the view synthesis model to generate an estimated transmittance; and

determining the visibility estimate from the estimated transmittances for the plurality of points.

14 . The method of claim 13 , wherein the visibility estimate is a mean of the estimated transmittances.

15 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one more computers to perform operations comprising:

maintaining a plurality of view synthesis models, wherein each view synthesis model is configured to receive an input specifying a camera viewpoint and to generate as output a synthesized image of the scene from the camera viewpoint;

obtaining an input specifying a new camera viewpoint;

selecting, based on the new camera viewpoint, a subset of the plurality of view synthesis models;

for each view synthesis model in the subset, processing a respective input specifying the new camera viewpoint to generate as output a synthesized image of the scene from the new camera viewpoint; and

combining the synthesized images generated by the view synthesis models in the subset to generate a final synthesized image of the scene from the new camera viewpoint, wherein the combining comprises:

determining a respective weight for each view synthesis model in the subset; and

generating the final synthesized image by interpolating between the synthesized images generated by the view synthesis models in the subset in accordance with the respective weights for the view synthesis models in the subset.

16 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations comprising:

maintaining a plurality of view synthesis models, wherein each view synthesis model is configured to receive an input specifying a camera viewpoint in the corresponding sub region and to generate as output a synthesized image of the scene from the camera viewpoint;

obtaining an input specifying a new camera viewpoint;

selecting, based on the new camera viewpoint, a subset of the plurality of view synthesis models;

for each view synthesis model in the subset, processing a respective input specifying the new camera viewpoint to generate as output a synthesized image of the scene from the new camera viewpoint; and

combining the synthesized images generated by the view synthesis models in the subset to generate a final synthesized image of the scene from the new camera viewpoint, wherein the combining comprises:

determining a respective weight for each view synthesis model in the subset; and

generating the final synthesized image by interpolating between the synthesized images generated by the view synthesis models in the subset in accordance with the respective weights for the view synthesis models in the subset.

17 . The system of claim 16 , wherein each view synthesis model has been trained on a respective set of training data that includes images captured from viewpoints within a corresponding sub-region of the scene.

18 . The system of claim 17 , wherein the view synthesis models have been trained independently on the respective sets of training data.

19 . The system of claim 16 , wherein selecting, based on the new camera viewpoint, a subset of the plurality of view synthesis models comprises:

selecting, for inclusion in the subset, each view synthesis model that corresponds to a respective sub-region of the scene that includes the new camera viewpoint.

20 . The system of claim 16 , wherein the operations further comprise:

determining, for each view synthesis model that corresponds to a respective sub-region of the scene that includes the new camera viewpoint, a respective visibility estimate that estimates a degree to which points along rays cast from the new viewpoint were visible in training images used to train the view synthesis model; and

removing, from the subset, any view synthesis model that has a respective visibility estimate that is below a visibility threshold.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 13, 2023
From: CASSER, VINCENT MICHAEL; KRETZSCHMAR, HENRIK; TANCIK, MATTHEW JUSTIN; PRADHAN, SABEEK MANI; MILDENHALL, BENJAMIN JOSEPH; SRINIVASAN, PRATUL PREETI; BARRON, JONATHAN TILTON
To: WAYMO LLC
Reel/Frame 062716/0328 →
Continuity (2)
Provisional Application 63285980 · Dec 3, 2021
Related Publication 20230177822A1 · Jun 8, 2023
References Cited (82)
US 11734792B2 · Yun · 2023 [cited by examiner]
US 12131422B2 · Zhuang · 2024 [cited by examiner]
US 20190258907A1 · Rezende et al. · 2019 [cited by applicant]
US 20190320164A1 · Salahieh · 2019 [cited by examiner]
US 20230154104A1 · Zhuang · 2023 [cited by examiner]
CN 110168560A · 2019 [cited by applicant]
CN 113014906A · 2021 [cited by applicant]
Agarwal et al., “Building rome in a day,” Communications of the ACM, Oct. 2011, pp. 105-112. [cited by applicant]
Amini et al., “Learning robust control policies for end-to-end autonomous driving from data-driven simulation,” IEEE Robotics and Automation Letters, Apr. 2020, 5(2):1143-1150. [cited by applicant]
Barron et al., “Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields,” CoRR, Mar. 24, 2021, arxiv.org/abs/2103.13415, 19 pages. [cited by applicant]
bmild.github.io/fourfeat/index.html [online], “Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains,” Jan. 24, 2021, retrieved on Jun. 2, 2023, retrieved from URL<https://bmild.github.… [cited by applicant]
Bojanowski et al., “Optimizing the latent space of generative networks,” CoRR, Jul. 18, 2017, arXiv:1707.05776, 10 pages. [cited by applicant]
Brualla et al., “NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections,” CoRR, Aug. 5, 2020, arxiv.org/abs/2008.02268, 15 pages. [cited by applicant]
Buehler et al., “Unstructured lumigraph rendering,” Computer graphics and interactive techniques, Aug. 2001, pp. 425-432. [cited by applicant]
Caesar et al., “nuScenes: A multimodal dataset for autonomous driving,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 11621-11631. [cited by applicant]
Chang et al., Argoverse: 3D tracking and forecasting with rich maps, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8748-8757. [cited by applicant]
Chen et al., “Geosim: Realistic video simulation via geometry-aware composition for self-driving,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 7230-7240. [cited by applicant]
Cheng et al., “Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 12475-12485. [cited by applicant]
Cordts et al., “The cityscapes dataset for semantic urban scene understanding,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3213-3223. [cited by applicant]
Deng et al., “Depth-supervised NeRF: Fewer Views and Faster Training for Free,” CoRR, Jul. 6, 2021, arxiv.org/abs/2107.02791, 13 pages. [cited by applicant]
Devaranjan et al., “Meta-sim2: Unsupervised learning of scene structure for synthetic data generation,” Computer Vision ECCV, 2020, 26 pages. [cited by applicant]
Dosovitskiy et al., “Carla: An open urban driving simulator,” Proceedings of the 1st Annual Conference on Robot Learning, 2017, 78:1-16. [cited by applicant]
Du et al., “The unmanned aerial vehicle benchmark: Object detection and tracking,” Proceedings of the European Conference on Computer Vision, 2018, pp. 370-386. [cited by applicant]
Extended European Search Report in European Appln. No. 22211253.4, dated May 9, 2023, 9 pages. [cited by applicant]
Flynn et al., “Deepstereo: Learning to predict new views from the world's imagery,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5515-5524. [cited by applicant]
Fruh et al., “An automated method for large-scale, ground-based city model acquisition,” IJCV, Oct. 2004, 60:5-24. [cited by applicant]
Furukawa et al., “Accurate, dense, and robust multi-view stereopsis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Aug. 2010, 32(8):1362-1376. [cited by applicant]
Furukawa et al., “Towards internet-scale multi-view stereo,” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Jun. 13-18, 2010, 8 pages. [cited by applicant]
Gaidon et al., “Virtual worlds as proxy for multi-object tracking analysis,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4340-4349. [cited by applicant]
Garbin et al., “Fastnerf: High-fidelity neural rendering at 200FPS,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 14346-14355. [cited by applicant]
Geiger et al., “Are we ready for autonomous driving? the kitti vision benchmark suite,” 2012 IEEE Conference on Computer Vision and Pattern Recognition, 8 pages. [cited by applicant]
Haklay et al., “Openstreetmap: Usergenerated street maps,” IEEE Pervasive Computing, Oct.-Dec. 2008, 7(4):12-18. [cited by applicant]
Hedman et al., “Baking neural radiance fields for real-time view synthesis,” CoRR, Mar. 26, 2021, arxiv.org/abs/2103.14645, 15 pages. [cited by applicant]
Hedman et al., “Deep blending for free-viewpoint image-based rendering,” ACM Transactions on Graphics, Dec. 4, 2018, 37(6):1-15. [cited by applicant]
Kar et al., “Meta-sim: Learning to generate synthetic datasets,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 4551-4560. [cited by applicant]
Kazhdan et al., “Screened poisson surface reconstruction,” ACM Transactions on Graphics, Jul. 4, 2013, 32(3):1-13. [cited by applicant]
Kim et al., “Drivegan: Towards a controllable high-quality neural simulation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 5820-5829. [cited by applicant]
Kingma et al., “Adam: A method for stochastic optimization,” CoRR, Dec. 22, 2014, arxiv.org/abs/1412.6980, 15 pages. [cited by applicant]
Kopf et al., “First-person hyper-lapse videos,” ACM Transactions on Graphics, Jul. 27, 2014, 33(4):1-10. [cited by applicant]
Kopf et al., “Street slide: browsing street level imagery,” ACM Transactions on Graphics, Jul. 26, 2010, 4(96):1-8. [cited by applicant]
Li et al., “AADS: Augmented autonomous driving simulation using data-driven algorithms,” Science Robotics, Mar. 27, 2019, 4(28):eaaw0863. [cited by applicant]
Li et al., “Modeling and recognition of landmark image collections using iconic scene graphs,” European Conference on Computer Vision, 2008, pp. 427-440. [cited by applicant]
Lin et al., “BARF: Bundle-Adjusting Neural Radiance Fields,” CoRR, Apr. 13, 2021, arxiv.org/abs/2104.06405, 15 pages. [cited by applicant]
Liu et al., “Infinite nature: Perpetual view generation of natural scenes from a single image,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 14458-14467. [cited by applicant]
Liu et al., “Neural Sparse Voxel Fields,” CoRR, Jan. 6, 2021, arXiv:2007.11571v2, 22 pages. [cited by applicant]
Lombardi et al., “Neural volumes: Learning dynamic renderable volumes from images,” CoRR, Jun. 18, 2019, arxiv.org/abs/1906.07751, 14 pages. [cited by applicant]
Lowe, “Distinctive image features from scaleinvariant keypoints,” IJCV, Nov. 2004, 60:91-110. [cited by applicant]
Martin-Brualla et al., “Nerf in the wild: Neural radiance fields for unconstrained photo collections,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 7210-7219. [cited by applicant]
matthewtancik.com [online], “Representing Scenes as Neural Radiance Fields for View Synthesis,” Mar. 20, 2020, retrieved on Jun. 2, 2023, retrieved from URL<https://www.matthewtancik.com/nerf/>, 10 pages. [cited by applicant]
Meng et al., “GNeRF: GAN-based Neural Radiance Field without Posed Camera,” CoRR, Mar. 29, 2021, arxiv.org/abs/2103.15606, 12 pages. [cited by applicant]
Meshry et al., “Neural rerendering in the wild,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6878-6887. [cited by applicant]
Mildenhall et al., “NeRF: representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, Aug. 3, 2020, 65(1):99-106. [cited by applicant]
Multiple View Geometry in Computer Vision, Cambridge University Press, Second Edition, 2003, Chapter 2, 48 pages. [cited by applicant]
Neff et al., “DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks,” CoRR, Mar. 4, 2021, arxiv.org/abs/2103.03231, 15 pages. [cited by applicant]
Ost et al., “Neural scene graphs for dynamic scenes,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2856-2865. [cited by applicant]
Park et al., “Nerfies: Deformable neural radiance fields,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 5865-5874. [cited by applicant]
Pollefeys et al., “Detailed real-time urban 3D reconstruction from video,” IJCV; Oct. 20, 2007, 25 pages. [cited by applicant]
Pulli et al., “View-based rendering: visualizing real objects from scanned rance and color data,” Rendering Techniques, Proceedings of the eurographics workshop, Jun. 16, 1997, 8:23-34. [cited by applicant]
Rebain et al., “DeRF: Decomposed radiance fields,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14153-14161. [cited by applicant]
Reiser et al., “KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 14335-14345. [cited by applicant]
Richter et al., “Enhancing photorealism enhancement,” CoRR, May 10, 2021, arXiv:2105.04619, 16 pages. [cited by applicant]
Richter et al., “Playing for data: Ground truth from computer games,” ECCV, Sep. 17, 2016, 16 pages. [cited by applicant]
Riegler et al., “Free view synthesis,” ECCV, Nov. 13, 2020, 17 pages. [cited by applicant]
Riegler et al., “Stable view synthesis,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 12216-12225. [cited by applicant]
Ros et al., “The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3234-3… [cited by applicant]
Schonberger et al., “Structure-from-motion revisited,” roceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4104-4113. [cited by applicant]
Shan et al., “The visual turing test for scene reconstruction,” 2013 International Conference on 3D Vision—3DV 2013, Jun. 29, 2013, pp. 25-32. [cited by applicant]
Snavely et al., “Photo tourism: Exploring photo collections in 3d,” Siggraph '06: ACM Siggraph 2006 Papers, Jul. 2006, pp. 835-846. [cited by applicant]
Sun et al., “Scalability in perception for autonomous driving: Waymo open dataset,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 2446-2454. [cited by applicant]
Takikawa et al., “Neural geometric level of detail: Real-time rendering with implicit 3D shapes,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 11358-11367. [cited by applicant]
Tancik et al., “Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains,” CoRR, Jun. 16, 2020, arxiv.org/abs/2006.10739, 24 pages. [cited by applicant]
Thrun, “Probabilistic robotics,” Communications of the ACM, 2002, 45(3):52-57. [cited by applicant]
Triggs et al., “Bundle adjustment—a modern synthesis. International workshop on vision algorithms,” Lecture Notes in Computer Science, 2000, 1883:298-372. [cited by applicant]
Wang et al., “NeRF—: Neural Radiance Fields Without Known Camera Parameters,” CoRR, Feb. 14, 2021, arxiv.org/abs/2102.07064, 17 pages. [cited by applicant]
Yang et al., “Learning object-compositional neural radiance field for editable scene rendering,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 13779-13788. [cited by applicant]
Yang et al., “Surfelgan: Synthesizing realistic sensor data for autonomous driving,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 11118-11127. [cited by applicant]
Yu et al., “Plenoctrees for real-time rendering of neural radiance fields,” CoRR, Mar. 25, 2021, arXiv:2103.14024, 18 pages. [cited by applicant]
Zhang et al., “Editable free-viewpoint video using a layered neural representation,” ACM Transactions on Graphics, Jul. 19, 2021, 40(4):1-18. [cited by applicant]
Zhang et al., “The unreasonable effectiveness of deep features as a perceptual metric,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 586-595. [cited by applicant]
Zhou et al., “Stereo magnification: Learning view synthesis using multiplane images,” CoRR, May 24, 2018, arXiv:1805.09817, 12 pages. [cited by applicant]
Zhu et al., “Very large-scale global SFM by distributed motion averaging,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4568-4577. [cited by applicant]
Office Action in Chinese Appln. No. 202211552177.3, mailed on Jun. 23, 2025, 17 pages (with English translation). [cited by applicant]