Estimating 3D scene representations of images
Methods and systems are disclosed for performing operations for estimating a 3D scene representation from one or multiple 2D images. The operations include: receiving one or multiple two-dimensional (2D) images representing a real-world environment; and generating, by a machine learning model, a three-dimensional (3D) scene representation of the 2D image, which explicitly (separately) defines the a 3D shape and appearance of the background as well as a 3D position, 3D shape and appearance of each object of the scene depicted in the set of images, where the machine learning model has been trained in an unsupervised approach from a dataset of images and their camera poses (e.g. without any manually labelled annotations, such as depth maps, segmentation masks, object poses).
1 . A method comprising:
receiving, by one or more processors, a set of two-dimensional (2D) images representing a real-world environment comprising a scene;
sampling a three-dimensional (3D) scene representation from a posterior distribution conditioned on a single or multiple sets of images drawn from a distribution different than a training distribution; and
generating, by a machine learning model, the 3D scene representation of the set of 2D images, where the 3D scene representation explicitly and separately defines a 3D shape and appearance of a background of the scene and a 3D position, 3D shape and appearance of each object of the scene depicted in the set of 2D images, where the machine learning model has been trained in an unsupervised approach from a dataset of images and their camera poses.
2 . The method of claim 1 , further comprising:
inferring latent 3D scene representation variables from one or multiple observed images with camera poses, using a generative model which reflects a physical process by which an image is produced, when a camera captures a scene containing multiple objects.
3 . The method of claim 2 , wherein the inferring is performed by a generative latent-variable model.
4 . The method of claim 3 , further comprising:
representing 3D positions of objects with a one-hot vector by categorical variables.
5 . The method of claim 4 , wherein the one-hot vector chooses from a set of plausible candidate locations to use as a center position of an object.
6 . The method of claim 1 , wherein the machine learning model comprises a plurality of networks including:
a first network that maps a set of Gaussian latents describing a 3D scene to a given 3D scene representation; and
a second network that renders a 2D image for the given 3D scene representation, wherein the second network is configured to map points in canonical space of the given 3D scene representation to a color and density.
7 . The method of claim 1 , wherein the machine learning model generates a first probability of a given 2D image given a particular camera view, and wherein the machine learning model generates a second probability of a composition of objects and background in a given scene.
8 . The method of claim 7 , wherein the machine learning model comprises a plurality of parameterized components comprising:
a first component that represents a 3D object as a function from position to color and density;
a second component that represents a 3D background as a function from position to color and density; and
a third component that maps global scene latent Gaussian embedding that represents a composition of a scene to parameters of the 3D object and background latent embeddings comprising a given 3D scene representation.
9 . The method of claim 8 , wherein the machine learning model is trained by performing training operations comprising:
obtaining a first encoder that encodes an observed image and viewpoint to posterior parameters comprising a particular 3D scene representation of the observed image and viewpoint; and
obtaining a second encoder that encodes the particular 3D scene representation to a particular global scene latent embedding.
10 . The method of claim 9 , wherein the training operations comprise:
accessing a dataset of images comprising a plurality of views of a plurality of scenes; and
performing a first training stage comprising training the machine learning model to reconstruct a first of the images associated with a first of the plurality of views through object-level latent space embedding while disregarding scene-level latent embedding.
11 . The method of claim 10 , further comprising:
after the first training stage has converged, performing a second training stage comprising a probability of the object-level latent space embedding given the scene-level latent embedding.
12 . The method of claim 11 , wherein posterior distributions on Gaussian latents variables comprising background, shape, and color, are predicted and comprise diagonal Gaussians parametrized by mean and log-variance.
13 . The method of claim 12 , and wherein the posterior parameters for component positions comprise Gumbel-Softmax parametrized by logits.
14 . The method of claim 1 , further comprising displaying a virtual element associated with an augmented reality or virtual reality experience on a user device within a video comprising the set of 2D images based on the 3D scene representation of the real-world environment.
15 . A system comprising:
at least one processor of a user device; and
a memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receiving a set of two-dimensional (2D) images representing a first view of a real-world environment comprising a scene;
sampling a three-dimensional (3D) scene representation from a posterior distribution conditioned on a single or multiple sets of images drawn from a distribution different than a training distribution; and
generating, by a machine learning model, the 3D scene representation of the set of 2D images, where the 3D scene representation explicitly and separately defines a 3D shape and appearance of a background of the scene and a 3D position, 3D shape and appearance of each object of the scene depicted in the set of 2D images, where the machine learning model has been trained in an unsupervised approach from a dataset of images and their camera poses.
16 . A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving a set of two-dimensional (2D) images representing a first view of a real-world environment comprising a scene;
sampling a three-dimensional (3D) scene representation from a posterior distribution conditioned on a single or multiple sets of images drawn from a distribution different than a training distribution; and
generating, by a machine learning model, the 3D scene representation of the set of 2D images, where the 3D scene representation explicitly and separately defines a 3D shape and appearance of a background of the scene and a 3D position, 3D shape and appearance of each object of the scene depicted in the set of 2D images, where the machine learning model has been trained in an unsupervised approach from a dataset of images and their camera poses.
17 . The non-transitory computer-readable storage medium of claim 16 , the operations further comprising:
sampling 3D scene representation from a posterior distribution conditioned on a single or multiple sets of images drawn from a distribution different than a training distribution.
18 . The non-transitory computer-readable storage medium of claim 17 , the operations further comprising:
inferring latent 3D scene representation variables from one or multiple observed images with camera poses, using a generative model which reflects a physical process by which an image is produced, when a camera captures a scene containing multiple objects.
19 . The non-transitory computer-readable storage medium of claim 18 , wherein the inferring is performed by a generative latent-variable model.
20 . The non-transitory computer-readable storage medium of claim 16 , wherein the machine learning model comprises one or more networks programmed to map a set of Gaussian latents describing a 3D scene to a given 3D scene representation and renders a 2D image for the given 3D scene representation, wherein the one or more networks are programmed to map points in canonical space of the given 3D scene representation to a color and density.