IP Library › Granted Patent US 12,481,893
Granted Patent B2
US 12,481,893 · App. 18/164,021 · Granted Nov 25, 2025

Scene understanding and generation using neural networks

Inventors: Danilo Jimenez Rezende (London, GB); Seyed Mohammadali Eslami (London, GB); Karol Gregor (London, GB); Frederic Olivier Besse (London, GB)
Assignee: GDM Holding LLC
G06N3/084G06F18/214G06N3/044G06N3/045G06T7/70G06T7/90G06T11/00G06V10/82G06V20/00G06V20/41G06V30/274G06T2207/10004G06T2207/10016G06T2207/20081G06T2207/20084G06T2207/20132G06T2207/30244
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,481,893
App. No.
18/164,021
Granted
Nov 25, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for image rendering. In one aspect, a method comprises receiving a plurality of observations characterizing a particular scene, each observation comprising an image of the particular scene and data identifying a location of a camera that captured the image. In another aspect, the method comprises receiving a plurality of observations characterizing a particular video, each observation comprising a video frame from the particular video and data identifying a time stamp of the video frame in the particular video. In yet another aspect, the method comprises receiving a plurality of observations characterizing a particular image, each observation comprising a crop of the particular image and data characterizing the crop of the particular image. The method processes each of the plurality of observations using an observation neural network to determine a numeric representation as output.

Claims (99)

1 . A method performed by one or more computers for generating a numerical representation of a scene, the method comprising:

receiving a plurality of observations characterizing the scene, each observation comprising: (i) an image of the scene, and (ii) data identifying a location of a camera that captured the image;

processing each of the plurality of observations using an observation neural network to generate a respective lower dimensional representation of each of the plurality of observations;

generating the numerical representation of the scene by combining the respective lower dimensional representations of each of the plurality of observations; and

providing the numerical representation of the scene for use in characterizing content of the scene, wherein the observation neural network has been jointly trained with a generator neural network that is configured to:

receive data identifying a new camera location;

process a network input to the generator neural network, the network input comprising: (i) the data identifying the new camera location, and (ii) the numerical representation of the scene; and

generate, by the generator neural network, a network output that comprises a new image of the scene that represents the view of the scene from the camera at the new camera location.

2 . The method of claim 1 , wherein generating the numerical representation of the scene by combining the lower dimensional representations of the plurality of observations comprises:

summing the lower dimensional representations of the plurality of observations.

3 . The method of claim 1 , wherein generating the numerical representation of the scene by combining the lower dimensional representations of the plurality of observations comprises:

processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the scene.

4 . The method of claim 3 , wherein processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the scene comprises:

processing the respective lower dimensional representation corresponding to each of the plurality of observations using a recurrent neural network;

wherein numerical representation of the scene is based on a hidden state of the recurrent neural network after the recurrent neural network has processed the respective lower dimensional representation corresponding to each of the plurality of observations.

5 . The method of claim 1 , wherein for one or more of the plurality of observations, the data identifying the location of the camera that captured the observation defines a three-dimensional position, yaw, and pitch of the camera.

6 . The method of claim 1 , wherein the observation neural network comprises one or more convolutional neural network layers.

7 . The method of claim 1 , wherein providing the numerical representation of the scene for use in characterizing the content of the scene comprises:

computationally rendering the new image of the scene that represents the view of the scene from the camera at the new camera location using the numerical representation of the scene.

8 . The method of claim 7 , wherein computationally rendering the new image of the scene that represents a view of the scene from the camera at the new camera location using the numerical representation of the scene comprises:

receiving data identifying the new camera location;

processing, using the generator neural network, a network input to the generator neural network, the network input comprising: (i) the data identifying the new camera location, and (ii) the numerical representation of the scene; and

generating, by the generator neural network and in response to processing the network input, a network output that comprises the new image of the scene that represents the view of the scene from the camera at the new camera location.

9 . The method of claim 8 , wherein processing the network input using the generator neural network to generate the network output comprises:

at each of a plurality of time steps:

sampling one or more latent variables for the time step; and

updating a hidden state of the generator neural network as of the time step by processing the hidden state, the sampled latent variables, the numerical representation of the scene, and the data identifying the new camera location; and

after a last time step in the plurality of time steps:

generating the new image of the scene from the updated hidden state of the generator neural network.

10 . The method of claim 8 , wherein processing the network input using the generator neural network to generate the network output comprises:

processing the network input, using the generator neural network, to generate respective pixel sufficient statistics for each pixel in the new image of the scene; and

sampling a respective color value for each pixel in the new image of the scene using the pixel sufficient statistics for the pixel.

11 . The method of claim 8 , wherein the generator neural network and the observation neural network have been trained jointly with a posterior neural network that is configured to, during the training, receive a plurality of training observations and a target observation and generate a posterior output that defines a distribution over one or more latent variables.

12 . A method performed by one or more computers for generating a numerical representation of a video, the method comprising:

receiving a plurality of observations characterizing the video, each observation comprising: (i) a video frame of the video, and (ii) data identifying a time stamp of the video frame of the video;

processing each of the plurality of observations using an observation neural network to generate a respective lower dimensional representation of each of the plurality of observations;

generating the numerical representation of the video by combining the respective lower dimensional representations of each of the plurality of observations; and

providing the numerical representation of the video for use in characterizing the video, wherein the observation neural network has been jointly trained with a generator neural network that is configured to:

receive data identifying a new time stamp;

process a network input to the generator neural network, the network input comprising: (i) the data identifying the new time stamp, and (ii) the numerical representation of the video; and

generate, by the generator neural network, a network output that comprises a new video frame at the new time stamp.

13 . The method of claim 12 , wherein generating the numerical representation of the video by combining the lower dimensional representations of the plurality of observations comprises:

summing the lower dimensional representations of the plurality of observations.

14 . The method of claim 12 , wherein generating the numerical representation of the video by combining the lower dimensional representations of the plurality of observations comprises:

processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the video.

15 . The method of claim 14 , wherein processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the video comprises:

processing the respective lower dimensional representation corresponding to each of the plurality of observations using a recurrent neural network;

wherein numerical representation of the video is based on a hidden state of the recurrent neural network after the recurrent neural network has processed the respective lower dimensional representation corresponding to each of the plurality of observations.

16 . The method of claim 12 , wherein for one or more of the plurality of observations, the time stamp of the video frame defines a time at which the video frame was captured.

17 . The method of claim 12 , wherein the observation neural network comprises one or more convolutional neural network layers.

18 . The method of claim 12 , wherein providing the numerical representation of the video for use in characterizing the video comprises:

computationally rendering the new video frame that represents the video frame in the video at the new time stamp using the numerical representation of the video.

19 . The method of claim 18 , wherein computationally rendering a new video frame that represents a video frame in the video at a new time stamp using the numerical representation of the video comprises:

receiving data identifying the new time stamp;

processing, using the generator neural network, a network input to the generator neural network, the network input comprising: (i) the data identifying the new time stamp, and (ii) the numerical representation of the video; and

generating, by the generator neural network and in response to processing the network input, a network output that comprises the new video frame at the new time stamp.

20 . The method of claim 19 , wherein processing the network input using the generator neural network to generate the network output comprises:

at each of a plurality of time steps:

sampling one or more latent variables for the time step; and

updating a hidden state of the generator neural network as of the time step by processing the hidden state, the sampled latent variables, the numerical representation of the video, and the data identifying the new time stamp; and

after a last time step in the plurality of time steps:

generating the new video frame from the updated hidden state of the generator neural network.

21 . The method of claim 19 , wherein processing the network input using the generator neural network to generate the network output comprises:

processing the network input, using the generator neural network, to generate respective pixel sufficient statistics for each pixel in the new video frame; and

sampling a respective color value for each pixel in the new video frame using the pixel sufficient statistics for the pixel.

22 . The method of claim 19 , wherein the generator neural network and the observation neural network have been trained jointly with a posterior neural network that is configured to, during the training, receive a plurality of training observations and a target observation and generate a posterior output that defines a distribution over one or more latent variables.

23 . A method performed by one or more computers for generating a numerical representation of an image, the method comprising:

receiving a plurality of observations characterizing the image, each observation comprising: (i) a crop of the image, and (ii) data identifying a location and size of the crop in the image;

processing each of the plurality of observations using an observation neural network to generate a respective lower dimensional representation of each of the plurality of observations;

generating the numerical representation of the image by combining the respective lower dimensional representations of each of the plurality of observations; and

providing the numerical representation of the image for use in characterizing content of the image, wherein the observation neural network has been jointly trained with a generator neural network that is configured to:

receive data identifying a new crop location and a new crop size of a new crop of the image;

process a network input to the generator neural network, the network input comprising: (i) the data identifying the new crop location and the new crop size, and (ii) the numerical representation of the image; and

generate, by the generator neural network, a network output that comprises the new crop of the image at the new crop location and with the new crop size.

24 . The method of claim 23 , wherein generating the numerical representation of the image by combining the lower dimensional representations of the plurality of observations comprises:

summing the lower dimensional representations of the plurality of observations.

25 . The method of claim 23 , wherein generating the numerical representation of the image by combining the lower dimensional representations of the plurality of observations comprises:

processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the image.

26 . The method of claim 25 , wherein processing the lower dimensional representations of the plurality of observations using one or more neural network layers to generate the numerical representation of the image comprises:

processing the respective lower dimensional representation corresponding to each of the plurality of observations using a recurrent neural network;

wherein numerical representation of the image is based on a hidden state of the recurrent neural network after the recurrent neural network has processed the respective lower dimensional representation corresponding to each of the plurality of observations.

27 . The method of claim 23 , wherein for one or more of the plurality of observations, the data identifying the location and size of the crop in the image includes coordinates of a vertices of a bounding box in the image.

28 . The method of claim 23 , wherein the observation neural network comprises one or more convolutional neural network layers.

29 . The method of claim 23 , wherein providing the numerical representation of the image for use in characterizing the content of the image comprises:

computationally rendering the new crop of the image that represents the crop of the image at the new crop location and having the new crop size using the numerical representation of the image.

30 . The method of claim 29 , wherein computationally rendering the new crop of the image using the numerical representation of the image comprises:

receiving data identifying the new crop location and the new crop size of the new crop;

processing, using the generator neural network, a network input to the generator neural network, the network input comprising: (i) the data identifying the new crop location and the new crop size, and (ii) the numerical representation of the image; and

generating, by the generator neural network and in response to processing the network input, a network output that comprises the new crop of the image at the new crop location and with the new crop size.

31 . The method of claim 30 , wherein processing the network input using the generator neural network to generate the network output comprises:

at each of a plurality of time steps:

sampling one or more latent variables for the time step; and

updating a hidden state of the generator neural network as of the time step by processing the hidden state, the sampled latent variables, the numerical representation of the image, and the data identifying the new crop location; and

after a last time step in the plurality of time steps:

generating the new crop of the image from the updated hidden state of the generator neural network.

32 . The method of claim 30 , wherein processing the network input using the generator neural network to generate the network output comprises:

processing the network input, using the generator neural network, to generate respective pixel sufficient statistics for each pixel in the new crop of the image; and

sampling a respective color value for each pixel in the new crop of the image using the pixel sufficient statistics for the pixel.

33 . The method of claim 30 , wherein the generator neural network and the observation neural network have been trained jointly with a posterior neural network that is configured to, during the training, receive a plurality of training observations and a target observation and generate a posterior output that defines a distribution over one or more latent variables.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 3, 2023
From: REZENDE, DANILO JIMENEZ; ESLAMI, SEYED MOHAMMADALI; GREGOR, KAROL; BESSE, FREDERIC OLIVIER
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 062586/0259 →
Continuity (5)
Continuation 16403278 · May 3, 2019
Continuation PCTIB2017056903 · Nov 4, 2017
Provisional Application 62540817 · Aug 3, 2017
Provisional Application 62418144 · Nov 4, 2016
Related Publication 20230177343A1 · Jun 8, 2023
References Cited (79)
US 10442593B2 · Jowett · 2019 [cited by examiner]
US 20080240616A1 · Haering · 2008 [cited by applicant]
US 20090185745A1 · Momosaki · 2009 [cited by applicant]
US 20110149072A1 · McCormack · 2011 [cited by examiner]
US 20110181711A1 · Reid · 2011 [cited by applicant]
US 20160125572A1 · Byungin et al. · 2016 [cited by applicant]
US 20170185872A1 · Chakraborty · 2017 [cited by applicant]
CN 105868829 · 2016 [cited by applicant]
CN 106056213 · 2016 [cited by applicant]
EP 3229459 · 2017 [cited by applicant]
JP 2016082350 · 2016 [cited by applicant]
JP 2019523597 · 2019 [cited by applicant]
KR 20160053612 · 2016 [cited by applicant]
KR 1020160087222 · 2016 [cited by applicant]
WO WO2016088437 · 2017 [cited by applicant]
WO WO2018015791 · 2018 [cited by applicant]
Office Action in Chinese Appln. No. 201780077072.8, dated Apr. 27, 2023, 17 pages (with English translation). [cited by applicant]
Office Action in Japanese Appln. No. 2021-101631, dated May 15, 2023, 5 pages (with English translation). [cited by applicant]
Eslami et al., “Supplementary Materials for: Neural scene representation and rendering,” Science, Jun. 15, 2018, 360:6394. [cited by applicant]
Eslami et al.: “Neural scene representation and rendering,” Science, Jun. 15, 2018, 360:6394. [cited by applicant]
Summons to Attend Oral Proceedings in European Appln. No. 17808162.6, dated Nov. 2, 2023, 14 pages. [cited by applicant]
Office Action in European Appln. 17808162.6, dated Sep. 4, 2024, 50 pages. [cited by applicant]
Office Action in Korean Appln. 10-2022-7036566, dated Jan. 12, 2024, 9 pages. [cited by applicant]
Agrawal et al, “Learning to see by moving,” Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 37-45. [cited by applicant]
Bruna et al, “Super-Resolution with Deep Convolutional Sufficient Statistics,” arXiv preprint arXiv:1511.05666, 2015, 17 pages. [cited by applicant]
Choy et al., “3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction,” Computer Vision—ECCV 2016 : 14th European Conference, 8:628-644. [cited by applicant]
Doersch et al, “Unsupervised visual representation learning by context prediction,” International Conference on Computer Vision, 2015, 9 pages. [cited by applicant]
Eslami & Williams, “A Generative Model for Parts-based Object Segmentation,” Advances in Neural Information Processing Systems 25, 2014, 9 pages. [cited by applicant]
Eslami et al, “Attend, infer, repeat: Fast scene understanding with generative models,” NIPS, 2016, 9 pages. [cited by applicant]
Gregor et al, “Draw: A Recurrent Neural Network for Image Generation,” International Conference on Machine Learning, 2015, 10 pages. [cited by applicant]
Gregor et al, “Towards conceptual compression,” Advances in Neural Information Processing Systems, 2016, pp. 3549-3557. [cited by applicant]
Heess et al, “Weakly Supervised Learning of Foreground-Background Segmentation Using Masked RBMs,” International Conference on Artificial Neural Networks, 2011, 8 pages. [cited by applicant]
Higgins et al, “Early visual concept learning with unsupervised deep learning,” CoRR, 12 pages. [cited by applicant]
Hinton et al, “The “wake-sleep” algorithm for unsupervised neural networks,” Science, 1995, 12 pages. [cited by applicant]
Hinton et al, “Transforming Auto-encoders,” International Conference on Artificial Neural Networks, 2011, 8 pages. [cited by applicant]
Hinton, “Training Products of Experts by Minimizing Contrastive Divergence,” Neural Computation, 2002, pp. 1771-1800. [cited by applicant]
Huang & Murphy, “Efficient inference in occlusion-aware generative models of images,” CoRR, 2015, 10 pages. [cited by applicant]
Jampani et al, “The Informed Sampler: A Discriminative Approach to Bayesian Inference in Generative Computer Vision Models,” Special Issue on Generative Models in Computer Vision and Medical Imaging, 2015, 20 pages. [cited by applicant]
JP Office Action in Japanese Appln. No. 2019-523597, dated Aug. 11, 2020, 7 pages (with English translation). [cited by applicant]
JP Office Action in Japanese Appln. No. 2019-523597, dated Mar. 15, 2021, 7 pages (with English translation). [cited by applicant]
Kalchbrenner et al, “Video pixel networks,” arXiv preprint arXiv:1610.00527, 2016, 16 pages. [cited by applicant]
Kingma et al, “Auto-Encoding Variational Bayes,” arXiv preprint arXiv:1312.6114, 2013, 14 pages. [cited by applicant]
KR Office Action in Korean Appln. No. 10-2019-7015932, dated Jun. 29, 2021, 11 pages (with English translation). [cited by applicant]
KR Office Action in Korean Appln. No. 10-2019-7015932, dated Nov. 30, 2020, 11 pages (with English translation). [cited by applicant]
Krizhevsky et al, “ImageNet Classification with Deep Convolutional Neural Networks,” Advances in Neural Information Processing Systems 25, 2012, 9 pages. [cited by applicant]
Kulkarni et al, “Deep Convolutional Inverse Graphics Network,” Advances in Neural Information Processing Systems 28, 2015, 9 pages. [cited by applicant]
Kulkarni et al, “Picture: A probabilistic programming language for scene perception,” Computer Vision and Pattern Recognition, 2015, 10 pages. [cited by applicant]
Le Roux et al, “Learning a generative model of images by factoring appearance and shape,” International Conference on Artificial Neural Networks, 2011, pp. 593-650. [cited by applicant]
Long et al, “Fully convolutional networks for semantic segmentation,” 2015, 10 pages. [cited by applicant]
Loper & Black, “OpenDR: An Approximate Differentiable Renderer,” European Conference on Computer Vision, 2014, pp. 154-169. [cited by applicant]
Mansinghka et al, “Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs,” Advances in Neural Information Processing Systems 26, 2013, 9 pages. [cited by applicant]
Memisevic et al, “Learning to represent spatial transformations with factored higher-order boltzmann machines,” Neural Computation, 2010, 15 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201780077072.8, dated Oct. 20, 2022, 16 pages (with English translation). [cited by applicant]
Office Action in European Appln. 17808162.6, dated Aug. 25, 2021, 9 pages. [cited by applicant]
Office Action in Japanese Appln. No. 2021-101631, dated Aug. 15, 2022, 8 pages (with English translation). [cited by applicant]
Office Action in Japanese Appln. No. 2021-101631, dated Dec. 19, 2022, 4 pages (with English translation). [cited by applicant]
Office Action in Korean Appln. 10-2019-7015932, dated Oct. 28, 2021, 3 pages. [cited by applicant]
Office Action in Korean Appln. 10-2019-7037705, dated Aug. 28, 2022, 4 pages. [cited by applicant]
Patraucean et al, “Spatio-temporal video autoencoder with differentiable memory,” arXiv preprint arXiv:1511.06309, 13 pages. [cited by applicant]
PCT International Preliminary Report on Patentability in International Appln. No. PCT/IB2017/056903, dated Feb. 18, 2019, 10 pages. [cited by applicant]
PCT International Search Report in International Appln. No. PCT/IB2017/056903, dated Feb. 23, 2018, 19 pages. [cited by applicant]
Pero et al, “Bayesian geometric modeling of indoor scenes,” Computer Vision and Pattern Recognition(CVPR), 2012, pp. 2719-2726. [cited by applicant]
Radford et al., “Unsupervised representation learning with deep convolutional generative adversarial networks,” 4th International Conference on Learning Representations, May 2016, 16 pages. [cited by applicant]
Ren et al, “Faster R-CNN: Towards real-time object detection with region proposal networks,” Advances in Neural Information Processing Systems 28, 2015, 9 pages. [cited by applicant]
Rezende et al, “Stochastic Backpropagation and Approximate Inference in Deep Generative Models,” International Conference on Machine Learning, 2014, 14 pages. [cited by applicant]
Rezende et al, “Unsupervised Learning of 3D Structure from Images,” Advances in Neural Information Processing Systems, 2016, pages 10 pages. [cited by applicant]
Rezende, et al, “One-Shot Generalization in Deep Generative Models,” arXiv preprint arXiv:1603.05106, 2016, 10 pages. [cited by applicant]
Salakhutdinov & Hinton, “Deep Boltzmann Machines,” International Conference on Artificial Intelligence and Statistics, 2009, 8 pages. [cited by applicant]
Sohn, et al., “Learning Structured Output Representation using Deep Conditional Generative Models,” Advances in Neural Information Processing Systems 28 (NIPS 2015), 2015, 9 pages. [cited by applicant]
Srivastava et al, “Unsupervised learning of video representations using lstms,” arXiv preprint arXiv:1502.04681, 10 pages. [cited by applicant]
Tang et al, “Learning Generative Models With Visual Attention,” Advances in Neural Information Processing Systems 27, 2014, 9 pages. [cited by applicant]
Tatarchenko et al, “Multi-view 3d models from single images with a convolutional network,” European Conference on Computer Vision, 2016, 20 pages. [cited by applicant]
Tu et al, “Image parsing: Unifying segmentation, detection, and recognition,” International Journal of computer vision, 2005, pp. 63(2):113-140. [cited by applicant]
Vondrick et al., “Anticipating the future by watching unlabeled video,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 98-106. [cited by applicant]
Zhao & Zhu, “Image parsing with stochastic scene grammar,” NIPS, 2011, pp. 73-81. [cited by applicant]
Zhou & Berg, “Learning temporal transformations from time-lapse videos,” European Conference on Computer Vision, 2016, 16 pages. [cited by applicant]
Zhou et al, “Learning deep features for scene recognition using places database,” Advances in Neural Information Processing Systems 27, 2014, 9 pages. [cited by applicant]
Zhu & Mumford, “A Stochastic Grammar of Images,” Foundations and Trends in Computer Graphics and Vision, 2006, 62 pages. [cited by applicant]
Allowance of Patent in Korean Appln. No. 10-2022-7036566, dated Sep. 25, 2024, 5 pages (with English translation). [cited by applicant]