View synthesis robust to unconstrained image data
Provided are systems and methods for synthesizing novel views of complex scenes (e.g., outdoor scenes). In some implementations, the systems and methods can include or use machine-learned models that are capable of learning from unstructured and/or unconstrained collections of imagery such as, for example, “in the wild” photographs. In particular, example implementations of the present disclosure can learn a volumetric scene density and radiance represented by a machine-learned model such as one or more multilayer perceptrons (MLPs).
1. A computing system to generate synthetic imagery of a scene, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned view synthesis model; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
accessing an existing training image that is associated with the scene, wherein the existing training image encodes a ground truth pixel color for a training pixel;
generating a static color based on descriptive data and the machine-learned view synthesis model, wherein the descriptive data is descriptive of a position within three-dimensional space that is associated with the scene;
generating a transient color based on the descriptive data and the machine-learned view synthesis model;
generating a synthetic pixel color for a synthetic pixel of a synthetized image that depicts the scene, wherein the synthetic pixel color is generated based on the static color, the transient color, and the machine-learned view synthesis model;
evaluating a loss function that compares the synthetic pixel color with a ground truth pixel color; and
training the machine-learned view synthesis model based on the loss function.
2. The computing system of claim 1 , wherein training the machine-learned view synthesis model includes modifying one or more parameter values for one or more parameters of the machine-learned view synthesis model based at least in part on the loss function.
3. The computing system of claim 1 , wherein generating the synthetic pixel color is further based on a static opacity and a transient opacity.
4. The computing system of claim 1 , wherein the operations further comprise:
generating a latent representation based on the descriptive data and the machine-learned view synthesis model;
generating the static color further based on the latent representation; and
generating the transient color further based on the latent representation.
5. The computing system of claim 1 , wherein the operations further comprise:
generating a static opacity based on the descriptive data and the machine-learned view synthesis model;
generating a transient opacity based on a latent representation and the machine-learned view synthesis model, wherein the latent representation is generated based on the descriptive data and the machine-learned view synthesis model; and
performing volumetric rendering to generate the synthetic color, and wherein generating the synthetic color is further based on the static opacity and the transient opacity.
6. The computing system of claim 5 , wherein the machine-learned view synthesis model includes a base portion and the operations further comprise:
generating the latent representation based on the base portion of the machine-learned view synthesis model processing the descriptive data; and
generating the static opacity based on the base portion of the machine-learned view synthesis model processing the descriptive data.
7. The computing system of claim 5 , wherein the machine-learned view synthesis model includes a static content portion that models static content within the scene and the operations further comprise:
generating the static color further based on that static portion of the machine-learned view synthesis models processing the latent representation.
8. The computing system of claim 5 , wherein the machine-learned view synthesis model includes a transient content portion that models transient content within the scene and the operations further comprise:
generating the transient opacity further based on the transient content portion of the machine-learned view synthesis model processing the latent representation; and
generating the transient color further based on the transient content portion of the machine-learned view synthesis model processing the latent representation.
9. The computing system of claim 8 , wherein:
said processing the latent representation with the transient content portion of the machine-learned view synthesis model further generates an uncertainty value; and
the loss function includes a pixel loss term that effects a negative correlation between loss magnitude and the uncertainty value.
10. The computing system of claim 8 , wherein the operations further comprise:
inputting an uncertainty embedding into the transient content portion of the machine-learned view synthesis model alongside the latent representation to generate the transient opacity and the transient color.
11. The computing system of claim 7 , wherein the operations further comprise:
inputting a generative embedding into the static content portion of the machine-learned view synthesis model alongside the latent representation to generate the static color.
12. The computing system of claim 11 , wherein the generative embedding is associated with the existing training image, and the operations further comprise:
modifying one or more values of the generative embedding based at least in part on the loss function.
13. The computing system of claim 1 , wherein the operations further comprise:
inputting one or more camera parameters into the base portion of the machine-learned view synthesis model alongside the position to generate the static opacity and the latent representation; and
modifying one or more of the camera parameters based at least in part on the loss function.
14. The computing system of claim 1 , wherein the computing system performs each of the operations for each respective pixel in the synthetic image.
15. The computing system of claim 1 , wherein the position comprises a location and an orientation.
16. The computing system of claim 1 , wherein the position comprises a new position that is not included in a training set that includes the existing training image.
17. The computing system of claim 1 , wherein the machine-learned view synthesis model comprises a respective multi-layer perceptron.
18. The computing system of claim 1 , wherein the position within three-dimensional space comprises a capture position associated with an existing training image and the operations further comprise:
obtaining the position within the three-dimensional space.
19. A computer-implemented method comprising:
accessing an existing training image that is associated with a scene, wherein the existing training image encodes a ground truth pixel color for a training pixel;
generating a static color based on descriptive data and the machine-learned view synthesis model, wherein the descriptive data is descriptive of a position within three-dimensional space that is associated with the scene;
generating a transient color based on the descriptive data and the machine-learned view synthesis model;
generating a synthetic pixel color for a synthetic pixel of a synthetized image that depicts the scene, wherein the synthetic pixel color is generated based on the static color, the transient color, and the machine-learned view synthesis model;
evaluating a loss function that compares the synthetic pixel color with a ground truth pixel color; and
training the machine-learned view synthesis model based on the loss function.
20. A computer-implemented method comprising:
obtaining a position within three-dimensional space that is associated with a scene;
generating a static color based on descriptive data and a machine-learned view synthesis model, wherein the descriptive data is descriptive of the position within three-dimensional space;
generating a transient color based on the descriptive data and the machine-learned view synthesis model;
performing volumetric rendering to generate to generate a synthetic pixel color for a synthetic pixel of a synthetized image, wherein the synthetic pixel color is generated based on the static color, the transient color, and the machine-learned view synthesis model.