Geometry-free neural scene representations through novel-view synthesis
Provided are machine learning models that generate geometry-free neural scene representations through efficient object-centric novel-view synthesis. In particular, one example aspect of the present disclosure provides a novel framework in which an encoder model (e.g., an encoder transformer network) processes one or more RGB images (with or without pose) to produce a fully latent scene representation that can be passed to a decoder model (e.g., a decoder transformer network). Given one or more target poses, the decoder model can synthesize images in a single forward pass. In some example implementations, because transformers are used rather than convolutional or MLP networks, the encoder can learn an attention model that extracts enough 3D information about a scene from a small set of images to render novel views with correct projections, parallax, occlusions, and even semantics, without explicit geometry.
1 . A computer-implemented method to generate novel views of scenes more efficiently, the method comprising:
obtaining, by a computing system comprising one or more computing devices, one or more input images that depict a scene;
generating, by the computing system, one or more image embeddings respectively for the one or more input images;
processing, by the computing system, the one or more image embeddings with a machine-learned encoder model to generate a scene embedding that represents the scene;
obtaining, by the computing system, ray data descriptive of one or more ray castings for a predicted image of the scene;
processing, by the computing system, the scene embedding and the ray data with a machine-learned decoder model to generate synthesized image data for the one or more ray castings for the predicted image of the scene, wherein the machine-learned decoder model comprises a self-attention model, and wherein processing the scene embedding and the ray data with the machine-learned decoder model comprises:
generating key and value data elements from the scene embedding;
generating query data elements from the ray data; and
processing the key, value, and query data elements with the machine-learned decoder model to generate the synthesized image data for the one or more ray castings for the predicted image of the scene; and
providing, by the computing system, the predicted image of the scene as an output.
2 . The computer-implemented method of claim 1 , wherein the machine-learned encoder model comprises a self-attention model.
3 . The computer-implemented method of claim 1 , wherein the machine-learned encoder model and the machine-learned decoder model have been jointly trained using a shared loss function.
4 . The computer-implemented method of claim 1 , wherein at least the machine-learned encoder model has been pre-trained using different images that depict a different scene.
5 . The computer-implemented method of claim 1 , wherein generating, by the computing system, the one or more image embeddings respectively for the one or more input images comprises processing, by the computing system, the one or more input images with a convolutional neural network to respectively generate the one or more image embeddings.
6 . The computer-implemented method of claim 1 , wherein generating, by the computing system, the one or more image embeddings respectively for the one or more input images comprises generating, by the computing system, one or more learned position embeddings for the one or more input images.
7 . The computer-implemented method of claim 1 , wherein the synthesized image data for each ray casting comprises color data for a pixel of the predicted image that corresponds to the ray casting.
8 . The computer-implemented method of claim 1 , wherein the one or more input images comprises a plurality of input images respectively captured at a plurality of different poses relative to the scene.
9 . The computer-implemented method of claim 1 , wherein the one or more input images comprise unposed images that have an unspecified pose relative to the scene.
10 . The computer-implemented method of claim 1 , further comprising:
evaluating, by the computing system, a loss function that compares the synthesized image data for the one or more ray castings to ground truth image data for the one or more ray castings; and
modifying, by the computing system, one or more values of one or more parameters of the machine-learned decoder model based at least in part on the loss function.
11 . The computer-implemented method of claim 1 , further comprising:
evaluating, by the computing system, a loss function that compares the synthesized image data for the one or more ray castings to ground truth image data for the one or more ray castings; and
modifying, by the computing system, one or more values of one or more parameters of both the machine-learned decoder model and the machine-learned encoder model based at least in part on the loss function.
12 . A computing system for generating scene-specific predicted imagery more efficiently, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a scene embedding generated by a machine-learned encoder model from one or more images of a scene, wherein the scene embedding represents the scene;
obtaining ray data descriptive of one or more ray castings for a predicted image of the scene;
processing the scene embedding and the ray data with a machine-learned decoder model to generate synthesized image data for the one or more ray castings for the predicted image of the scene, wherein the machine-learned decoder model comprises a self-attention model, and wherein processing the scene embedding and the ray data with the machine-learned decoder model comprises:
generating key and value data elements from the scene embedding;
generating query data elements from the ray data; and
processing the key, value, and query data elements with the machine-learned decoder model to generate the synthesized image data for the one or more ray castings for the predicted image of the scene; and
providing the predicted image of the scene as an output.
13 . The computing system of claim 12 , wherein machine-learned encoder model comprises a self-attention model.
14 . The computing system of claim 12 , wherein the machine-learned encoder model and the machine-learned decoder model have been jointly trained using a shared loss function.
15 . The computing system of claim 12 , wherein at least the machine-learned encoder model has been pre-trained using different images that depict a different scene.
16 . One or more non-transitory computer-readable media that store instructions that, when executed by a computing system, cause the computing system to perform operations, the operations comprising:
obtaining one or more input images that depict a scene;
generating one or more image embeddings respectively for the one or more input images;
processing the one or more image embeddings with a machine-learned encoder model to generate a scene embedding that represents the scene;
obtaining ray data descriptive of one or more ray castings for a predicted image of the scene;
processing the scene embedding and the ray data with a machine-learned decoder model to generate synthesized image data for the one or more ray castings for the predicted image of the scene, wherein the machine-learned decoder model comprises a self-attention model, and wherein processing the scene embedding and the ray data with the machine-learned decoder model comprises:
generating key and value data elements from the scene embedding;
generating query data elements from the ray data; and
processing the key, value, and query data elements with the machine-learned decoder model to generate the synthesized image data for the one or more ray castings for the predicted image of the scene;
evaluating a shared loss function that compares the synthesized image data for the one or more ray castings to ground truth image data for the one or more ray castings; and
modifying one or more values of one or more parameters of the machine-learned decoder model based at least in part on the loss function.
17 . The one or more non-transitory computer-readable of claim 16 , wherein the operations further comprise modifying one or more values of one or more parameters of the machine-learned encoder model based at least in part on the loss function.
18 . The one or more non-transitory computer-readable of claim 16 , wherein the machine-learned encoder model comprises a self-attention model.
19 . The one or more non-transitory computer-readable of claim 16 , wherein at least the machine-learned encoder model has been pre-trained using different images that depict a different scene.