Learning 2D texture mapping in volumetric neural rendering

1. A computer-implemented method comprising:

obtaining a plurality of images of a scene depicting at least one object;

determining a volume density of the scene using a scene geometry network to generate a 3D geometric representation of the object;

mapping 3D points of the scene to a 2D texture space using a texture mapping network, wherein the texture mapping network is trained on a cycle loss using an inverse texture mapping network that maps back from the 2D texture space to 3D points to train the texture mapping network and the inverse texture mapping network to enforce a cycle mapping between the 2D texture space and points on a surface of the scene; and

determining radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a texture network to generate a 3D appearance representation of the object.

2. The computer-implemented method of claim 1 , wherein the networks are trained on a rendering loss determined based on a difference between the radiance values determined by the texture network and ground truth radiance values obtained from a ground truth image.

3. The computer-implemented method of claim 1 , wherein the scene geometry network, texture mapping network, and texture network are multilayer perceptrons.

4. The computer-implemented method of claim 1 , wherein the 3D appearance representation of the object is a texture map.

5. The computer-implemented method of claim 4 , wherein the texture map represents a view-dependent texture or a fully relightable view-dependent reflectance.

6. The computer-implemented method of claim 4 , further comprising:

receiving a request to modify the 3D appearance representation of the object; and

modifying the texture map based on the request.

7. The computer-implemented method of claim 6 , wherein modifying the 2D texture space based on the request, further comprises:

combining the texture map with a new texture map included with the request to generate a modified texture map.

8. The computer-implemented method of claim 7 , further comprising:

receiving a viewpoint; and

generating a synthetic view of the scene based on the viewpoint using the 3D geometric representation of the object and the modified texture map.

9. A non-transitory computer readable storage medium including instructions stored thereon which, when executed by a processor, cause the processor to:

obtain a plurality of images of a scene depicting at least one object;

determine a volume density of the scene using a scene geometry network to generate a 3D geometric representation of the object;

map 3D points of the scene to a 2D texture space using a texture mapping network, wherein the texture mapping network is trained on a cycle loss using an inverse texture mapping network that maps back from the 2D texture space to 3D points to train the texture mapping network and the inverse texture mapping network to enforce a cycle mapping between the 2D texture space and points on a surface of the scene; and

determine radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a third neural texture network to generate a 3D appearance representation of the object.

10. The non-transitory computer readable storage medium of claim 9 , wherein the networks are trained on a rendering loss determined based on a difference between the radiance values determined by the texture network and ground truth radiance values obtained from a ground truth image.

11. The non-transitory computer readable storage medium of claim 10 , wherein the scene geometry network, texture mapping network, and texture network are multilayer perceptrons.

12. The non-transitory computer readable storage medium of claim 9 , wherein the representation of the 3D appearance representation of the object is a texture map.

13. The non-transitory computer readable storage medium of claim 12 , wherein the texture map represents a view-dependent texture or a fully relightable view-dependent reflectance.

14. The non-transitory computer readable storage medium of claim 12 , wherein the instructions, when executed, further cause the processor to:

receive a request to modify the 3D appearance representation of the object; and

modify the texture map based on the request.

15. The non-transitory computer readable storage medium of claim 14 , wherein to modify the 2D texture space based on the request, the instructions, when executed, further cause the processor to:

combine the texture map with a new texture map included with the request to generate a modified texture map.

16. The non-transitory computer readable storage medium of claim 15 , wherein the instructions, when executed, further cause the processor to:

receive a viewpoint; and

generate a synthetic view of the scene based on the viewpoint using the 3D geometric representation of the object and the modified texture map.

17. A computer-implemented method comprising:

receiving a request to edit a texture map generated by a volumetric neural rendering system including a plurality of neural networks, wherein the volumetric neural rendering system generates the texture map representing an appearance of an object in a scene separately from a 3D geometric representation of the object;

obtaining the texture map, wherein the texture map is generated at least in part using a texture mapping network, and wherein the texture mapping network is trained on a cycle loss using an inverse texture mapping network that maps back from a 2D texture space to 3D points to train the texture mapping network and the inverse texture mapping network to enforce a cycle mapping between the 2D texture space and points on a surface of the scene; and

generating a modified texture map based on the texture map and the request.

18. The computer-implemented method of claim 17 , wherein generating a modified texture map based on the texture map and the request, further comprises:

receiving a new texture map; and

combining the texture map with the new texture map to generate the modified texture map.