View synthesis robust to unconstrained image data
Provided are systems and methods for synthesizing novel views of complex scenes (e.g., outdoor scenes). In some implementations, the systems and methods can include or use machine-learned models that are capable of learning from unstructured and/or unconstrained collections of imagery such as, for example, “in the wild” photographs. In particular, example implementations of the present disclosure can learn a volumetric scene density and radiance represented by a machine-learned model such as one or more multilayer perceptrons (MLPs).
1. A computing system to generate synthetic imagery of a scene, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned view synthesis model that comprises a base portion, a static content portion that models static content within the scene, and a transient content portion that models transient content within the scene; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a position within three-dimensional space;
processing data descriptive of the position with the base portion of the machine-learned view synthesis model to generate a static opacity and a latent representation;
processing the latent representation with the static content portion of the machine-learned view synthesis model to generate a static color;
processing the latent representation with the transient content portion of the machine-learned view synthesis model to generate a transient opacity and a transient color; and
performing volumetric rendering to generate a synthetic pixel color for a synthetic pixel of a synthesized image from the static opacity, the static color, the transient opacity, and the transient color.
2. The computing system of claim 1 , wherein:
the position within three-dimensional space comprises a capture position associated with an existing training image; and
the operations further comprise:
evaluating a loss function that compares the synthetic pixel color with a ground truth pixel color for a training pixel including in the existing training image; and
modifying one or more parameter values for one or more parameters of the machine-learned view synthesis model based at least in part on the loss function.
3. The computing system of claim 2 , wherein:
said processing the latent representation with the transient content portion of the machine-learned view synthesis model further generates an uncertainty value; and
the loss function includes a pixel loss term that effects a negative correlation between loss magnitude and the uncertainty value.
4. The computing system of claim 1 , wherein the operations further comprise:
inputting an uncertainty embedding into the transient content portion of the machine-learned view synthesis model alongside the latent representation to generate the transient opacity and the transient color.
5. The computing system of claim 2 , wherein the operations further comprise:
inputting a generative embedding into the static content portion of the machine-learned view synthesis model alongside the latent representation to generate the static color.
6. The computing system of claim 5 , wherein:
the generative embedding is associated with the existing training image; and
the operations further comprise modifying one or more values of the generative embedding based at least in part on the loss function.
7. The computing system of claim 2 , wherein the operations further comprise:
inputting one or more camera parameters into the base portion of the machine-learned view synthesis model alongside the position to generate the static opacity and the latent representation; and
modifying one or more of the camera parameters based at least in part on the loss function.
8. The computing system of claim 1 , wherein the computing system performs each of the operations for each respective pixel in the synthetic image.
9. The computing system of claim 1 , wherein the position comprises a location and an orientation.
10. The computing system of claim 1 , wherein the position comprises a new position that is not included in a training set on which the machine-learned view synthesis model has trained.
11. The computing system of claim 1 , wherein each of the base portion, the static content portion, and the transient content portion of the machine-learned view synthesis model comprises a respective multi-layer perceptron.
12. A computer-implemented method for view synthesis with user-specifiable characteristics, the method comprising:
obtaining, by a computing system comprising one or more computing devices, a desired position within three-dimensional space and a user-specified generative embedding, wherein the generative embedding encodes one or more visual characteristics of a synthetic image to be generated;
processing, by the computing system, data descriptive of the position with a base portion of a machine-learned view synthesis model to generate an opacity and a latent representation;
processing, by the computing system, the latent representation and the generative embedding with a content portion of the machine-learned view synthesis model to generate a color; and
performing, by the computing system, volumetric rendering to generate a synthetic pixel color for a synthetic pixel of the synthesized image from the opacity and the color, wherein the synthesized image exhibits the one or more visual characteristics encoded by the generative embedding.
13. The computer-implemented method of claim 12 , wherein:
the opacity generated by the base portion comprises a static opacity; and
processing, by the computing system, the latent representation and the generative embedding with the content portion of the machine-learned view synthesis model to generate the color comprises:
processing, by the computing system, the latent representation and the generative embedding with a static content portion of the machine-learned view synthesis model to generate a static color; and
processing, by the computing system, the latent representation with a transient content portion of the machine-learned view synthesis model to generate a transient opacity and a transient color; and
performing, by the computing system, volumetric rendering to generate the synthetic pixel color for the synthetic pixel of the synthesized image from the opacity and the color comprises performing, by the computing system, volumetric rendering to generate the synthetic pixel color for the synthetic pixel of the synthesized image from the static opacity, the static color, the transient opacity, and the transient color.
14. The computer-implemented method of claim 13 , further comprising:
inputting, by the computing system, an uncertainty embedding into the transient content portion of the machine-learned view synthesis model alongside the latent representation to generate the transient opacity and the transient color.
15. The computer-implemented method of claim 12 , wherein the generative embedding comprises an interpolated embedding generated by interpolating respective image embeddings associated with two or more existing images selected by a user.
16. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:
obtaining, by the computing system, a position within three-dimensional space and a training embedding associated with an existing training image, wherein the training embedding encodes one or more visual characteristics of the existing training image;
processing, by the computing system, data descriptive of the position with a base portion of a machine-learned view synthesis model to generate an opacity and a latent representation;
processing, by the computing system, the latent representation and the training embedding with a content portion of the machine-learned view synthesis model to generate a color; and
performing, by the computing system, volumetric rendering to generate a synthetic pixel color for a synthetic pixel of a synthesized image from the opacity and the color;
evaluating a loss function that compares the synthetic pixel color with a ground truth pixel color for a training pixel including in the existing training image; and
modifying one or more values of the training embedding based at least in part on the loss function.
17. The one or more non-transitory computer-readable media of claim 16 , wherein:
the opacity generated by the base portion comprises a static opacity; and
processing, by the computing system, the latent representation and the training embedding with the content portion of the machine-learned view synthesis model to generate the color comprises:
processing, by the computing system, the latent representation and the training embedding with a static content portion of the machine-learned view synthesis model to generate a static color; and
processing, by the computing system, the latent representation with a transient content portion of the machine-learned view synthesis model to generate a transient opacity and a transient color; and
performing, by the computing system, volumetric rendering to generate the synthetic pixel color for the synthetic pixel of the synthesized image from the opacity and the color comprises performing, by the computing system, volumetric rendering to generate the synthetic pixel color for the synthetic pixel of the synthesized image from the static opacity, the static color, the transient opacity, and the transient color.
18. The one or more non-transitory computer-readable media of claim 17 , wherein:
said processing the latent representation with the transient content portion of the machine-learned view synthesis model further generates an uncertainty value; and
the loss function includes a pixel loss term that effects a negative correlation between loss magnitude and the uncertainty value.
19. The one or more non-transitory computer-readable media of claim 16 , wherein the operations further comprise:
inputting one or more camera parameters associated with a camera that captured the existing training image into the base portion of the machine-learned view synthesis model alongside the position to generate the opacity and the latent representation; and
modifying one or more of the camera parameters based at least in part on the loss function.
20. A computing system to generate synthetic imagery of a scene, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned view synthesis model that comprises a base portion and a static content portion that models static content within the scene, wherein the base portion and the static portion have been trained jointly with a transient content portion that models transient content within the scene; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a position within three-dimensional space;
processing data descriptive of the position with the base portion of the machine-learned view synthesis model to generate a static opacity and a latent representation;
processing the latent representation with the static content portion of the machine-learned view synthesis model to generate a static color; and
performing volumetric rendering to generate a synthetic pixel color for a synthetic pixel of a synthesized image from the static opacity and the static color.