Self-supervised depth for volumetric rendering regularization
An example method includes generating embeddings of image data that includes multiple images, where each image has a different viewpoints of a scene, generating a latent space and a decoder, wherein the decoder receives embeddings as input to generate an output viewpoint, for each viewpoint in the image data, determining a volumetric rendering view synthesis loss and a multi-view photometric loss, and applying an optimization algorithm to the latent space and the decoder over a number of epochs until the volumetric rendering view synthesis loss is within a volumetric threshold and the multi-view photometric loss is within a multi-view threshold.
1 . A method of training a latent space trainer for volumetric rendering, the method comprising:
generating embeddings of image data that includes multiple images by sampling values along a viewing ray to generate 3D points and Fourier encoding the sampled points, where each image has a different viewpoint of a scene;
generating a latent space and a decoder, wherein the decoder receives embeddings as input to generate an output viewpoint;
for each viewpoint in the image data, determining a volumetric rendering view synthesis loss and a multi-view photometric loss; and
applying an optimization algorithm to the latent space and the decoder over a number of epochs until the volumetric rendering view synthesis loss is within a volumetric threshold and the multi-view photometric loss is within a multi-view threshold.
2 . The method of claim 1 , wherein the optimization algorithm uses a Mean Square Error objective for the volumetric rendering view synthesis loss.
3 . The method of claim 1 , wherein the optimization algorithm is a gradient descent algorithm.
4 . The method of claim 1 , wherein the multi-view photometric loss is determined using a photometric objective.
5 . The method of claim 4 , wherein the photometric objective is determined by:
for each pixel of a target image of the image data, with a predicted depth ({circumflex over (d)}), generating, by a warping operation, projected coordinates with a predicted depth ({circumflex over (d)}′) in a context image;
generating a synthesized target image from the context image; and
determining a difference between the target image and the synthesized target image.
6 . The method of claim 5 , wherein the context image is generated by a transformation matrix.
7 . The method of claim 5 , wherein the difference between the target image and the synthesized target image is determined by a weighted structural similarity index.
8 . The method of claim 5 , wherein the synthesized target image is generated by applying grid sampling with bilinear interpolation to place information from the context image onto each target pixel of the synthesized target image based on the projected coordinates.
9 . The method of claim 5 , wherein pixels of the target image for determining the photometric objective are determined using strided ray sampling.
10 . A system for training a latent space trainer for volumetric rendering, the system comprising:
one or more processors;
a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to:
generate embeddings of image data that includes multiple images by sampling values along a viewing ray to generate 3D points and Fourier encoding the sampled points, where each image has a different viewpoint of a scene;
generate a latent space and a decoder, wherein the decoder receives embeddings as input to generate an output viewpoint;
for each viewpoint in the image data, determine a volumetric rendering view synthesis loss and a multi-view photometric loss; and
apply an optimization algorithm to the latent space and the decoder over a number of epochs until the volumetric rendering view synthesis loss is within a volumetric threshold and the multi-view photometric loss is within a multi-view threshold.
11 . The system of claim 10 , wherein the optimization algorithm uses a Mean Square Error objective for the volumetric rendering view synthesis loss.
12 . The system of claim 10 , wherein the optimization algorithm is a gradient descent algorithm.
13 . The system of claim 10 , wherein the multi-view photometric loss is determined using a photometric objective.
14 . The system of claim 13 , wherein the photometric objective is determined by:
for each pixel of a target image of the image data, with a predicted depth ({circumflex over (d)}), generating, by a warping operation, projected coordinates with a predicted depth ({circumflex over (d)}′) in a context image;
generating a synthesized target image from the context image; and
determining a difference between the target image and the synthesized target image.
15 . The system of claim 14 , wherein the context image is generated by a transformation matrix.
16 . The system of claim 14 , wherein the difference between the target image and the synthesized target image is determined by a weighted structural similarity index.
17 . The system of claim 14 , wherein the synthesized target image is generated by applying grid sampling with bilinear interpolation to place information from the context image onto each target pixel of the synthesized target image based on the projected coordinates.
18 . The system of claim 14 , wherein pixels of the target image for determining the photometric objective are determined using strided ray sampling.
19 . A tangible computer-readable medium comprising instructions that, when executed, cause a system to:
generate embeddings of image data that includes multiple images by sampling values along a viewing ray to generate 3D points and Fourier encoding the sampled points, where each image has a different viewpoint of a scene;
generate a latent space and a decoder, wherein the decoder receives embeddings as input to generate an output viewpoint;
for each viewpoint in the image data, determine a volumetric rendering view synthesis loss and a multi-view photometric loss; and
apply an optimization algorithm to the latent space and the decoder over a number of epochs until the volumetric rendering view synthesis loss is within a volumetric threshold and the multi-view photometric loss is within a multi-view threshold.
20 . The system of claim 19 , wherein the multi-view photometric loss is determined using a photometric objective.