Utilizing voxel feature transformations for view synthesis
Systems, methods, and non-transitory computer-readable media are disclosed for utilizing an encoder-decoder architecture to learn a volumetric 3D representation of an object using digital images of the object from multiple viewpoints to render novel views of the object. For instance, the disclosed systems can utilize patch-based image feature extraction to extract lifted feature representations from images corresponding to different viewpoints of an object. Furthermore, the disclosed systems can model view-dependent transformed feature representations using learned transformation kernels. In addition, the disclosed systems can recurrently and concurrently aggregate the transformed feature representations to generate a 3D voxel representation of the object. Furthermore, the disclosed systems can sample frustum features using the 3D voxel representation and transformation kernels. Then, the disclosed systems can utilize a patch-based neural rendering approach to render images from frustum feature patches to display a view of the object from various viewpoints.
1. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
generating lifted feature representations of an object from a plurality of two-dimensional (2D) digital images depicting the object from multiple viewpoints, each of the lifted feature representations corresponding to a particular viewpoint from the multiple viewpoints;
learning convolutional transformation kernels for the multiple viewpoints utilizing camera pose information from the plurality of 2D digital images;
generating transformed feature representations by applying the convolutional transformation kernels to the lifted feature representations;
generating, based on the plurality of 2D digital images depicting the object from the multiple viewpoints, a three-dimensional (3D) voxel feature representation for the object utilizing the transformed feature representations; and
rendering, for display, a 2D view depicting the object from a target viewpoint utilizing the 3D voxel feature representation and a convolutional transformation kernel for the target viewpoint.
2. The non-transitory computer-readable medium of claim 1 , wherein learning the convolutional transformation kernels comprises applying one or more 3D convolutional layers on the camera pose information for a given 2D digital image.
3. The non-transitory computer-readable medium of claim 1 , wherein generating the lifted feature representations comprises:
extracting feature maps from the plurality of 2D digital images utilizing a 2D U-Net; and
lifting features from the feature maps utilizing the camera pose information for the plurality of 2D digital images.
4. The non-transitory computer-readable medium of claim 1 , wherein generating the transformed feature representations comprises performing a 3D convolution operation between a respective convolutional transformation kernel and lifted feature representation pair.
5. The non-transitory computer-readable medium of claim 1 , wherein rendering the 2D view comprises:
utilizing the convolutional transformation kernel from the target viewpoint with the 3D voxel feature representation to sample a frustum feature; and
rendering the 2D view by utilizing a neural renderer on the frustum feature.
6. The non-transitory computer-readable medium of claim 1 , wherein generating the lifted feature representations comprises:
sampling a plurality of 2D digital image patches from the plurality of 2D digital images;
extracting feature maps from the plurality of 2D digital image patches; and
lifting features from the feature maps.
7. The non-transitory computer-readable medium of claim 1 , wherein generating the 3D voxel feature representation comprises recurrently and concurrently aggregating the transformed feature representations.
8. A computer-implemented method comprising:
generating lifted feature representations of an object from a plurality of two-dimensional (2D ) digital images depicting the object from multiple viewpoints, the lifted feature representations corresponding to particular viewpoints from the multiple viewpoints;
learning convolutional transformation kernels for the multiple viewpoints utilizing camera pose information from the plurality of 2D digital images;
generating transformed feature representations by applying the convolutional transformation kernels to the lifted feature representations;
generating, based on the plurality of 2D images depicting the object from the multiple viewpoints, a three-dimensional (3D) voxel feature representation for the object utilizing the transformed feature representations; and
rendering, for display, a 2D view depicting the object from a target viewpoint utilizing the 3D voxel feature representation and a convolutional transformation kernel for the target viewpoint.
9. The computer-implemented method of claim 8 , wherein generating the plurality of lifted feature representations comprises:
extracting feature maps from the plurality of 2D digital images; and
lifting features from the feature maps.
10. The computer-implemented method of claim 8 , wherein learning the convolutional transformation kernels comprises applying one or more 3D convolutional layers on the camera pose information for a given 2D digital image.
11. The computer-implemented method of claim 8 , wherein generating the 3D voxel feature representation comprises aggregating the transformed feature representations.
12. The computer-implemented method of claim 11 , wherein aggregating the transformed feature representations comprises pooling transformed feature representations.
13. The computer-implemented method of claim 11 , wherein aggregating the transformed feature representations comprises recurrently aggregating the transformed feature representations utilizing a gated recurrent unit.
14. The computer-implemented method of claim 8 , further comprising:
receiving a user selection of the target viewpoint; and
generate the 2D view from the 3D voxel feature representation of the object to depict the object from the target viewpoint utilizing a sampled frustum feature and the convolutional transformation kernel for the target viewpoint.
15. The computer-implemented method of claim 8 , wherein generating the transformed feature representations comprises performing a 3D convolution operation between a respective convolutional transformation kernel and lifted feature representation pair.
16. A system comprising:
one or more memory devices comprising a plurality of digital images depicting an object from multiple viewpoints; and
one or more processor devices configured to cause the system to:
generate lifted feature representations of an object from a plurality of two-dimensional (2D ) digital images depicting the object from multiple viewpoints, each of the lifted feature representations corresponding to a particular viewpoint from the multiple viewpoints;
learn convolutional transformation kernels for the multiple viewpoints utilizing camera pose information from the plurality of 2D digital images;
generate transformed feature representations by applying the convolutional transformation kernels to the lifted feature representations;
generate, based on the plurality of 2D digital images depicting the object from the multiple viewpoints, a three-dimensional (3D) voxel feature representation for the object utilizing the transformed feature representations; and
render, for display, a 2D view depicting the object from a target viewpoint utilizing the 3D voxel feature representation and a convolutional transformation kernel for the target viewpoint.
17. The system of claim 16 , wherein the one or more processor devices are further configured to generate the lifted feature representations by:
extracting feature maps from the plurality of 2D digital images; and
lifting features from the feature maps.
18. The system of claim 17 , wherein the one or more processor devices are further configured to:
sample a plurality of digital image patches from the plurality of 2D digital images; and
extract the feature maps from the plurality of digital image patches.
19. The system of claim 16 , wherein the one or more processor devices are configured to render the 2D view from the 3D voxel feature representation utilizing a sampled frustum feature and the convolutional transformation kernel associated with the target viewpoint.
20. The system of claim 16 , wherein the one or more processor devices are configured to learn the convolutional transformation kernels by applying one or more 3D convolutional layers on the camera pose information for a given 2D digital image.