Rendering textured surface using surface-rendering neural networks

1. A method comprising:

receiving, by a surface-rendering application, image metadata for rendering a texel of a textured surface, wherein the image metadata includes a coordinate identifying a location of the texel, an incoming radiance direction, an outgoing radiance direction, and a kernel size;

estimating, by the surface-rendering application and based on the image metadata, a light-reflectance value for the texel, wherein estimating the light-reflectance value includes:

applying an offset-prediction neural network to the coordinate and the incoming radiance direction to predict an offset coordinate of the texel;

inputting the offset coordinate to a data structure to determine a reflectance feature vector of the texel at the kernel size, wherein: (i) the data structure includes a set of neural textures; (ii) each neural texture of the set of neural textures includes a set of reflectance feature vectors at a respective kernel size; and (iii) each reflectance feature vector of the set of reflectance feature vectors includes one or more latent variables that encode visual characteristics of a respective texel; and

applying a decoder neural network to the reflectance feature vector, the incoming radiance direction, and the outgoing radiance direction to estimate a light-reflectance value for the texel; and

rendering, by the surface-rendering application and based on the light-reflectance value, the texel of the textured surface at the location, wherein the texel depicts at least part of the textured surface at an image scale that corresponds to the kernel size.

2. The method of claim 1 , wherein the data structure is a neural-texture pyramid, and wherein values of the one or more latent variables of a reflectance feature vector are learned using a backpropagation algorithm.

3. The method of claim 1 , wherein applying the offset-prediction neural network to the coordinate and the incoming radiance direction further includes:

querying a neural-offset texture using the coordinate to identify an offset feature vector of the coordinate;

applying the offset-prediction neural network to the offset feature vector and the incoming radiance direction to estimate a ray depth associated with the coordinate; and

applying an offset function to the ray depth to predict the offset coordinate.

4. The method of claim 1 , wherein querying the data structure further includes:

identifying, from a first neural texture of the set of neural textures, a first location within the first neural texture based on bilinear interpolation of the offset coordinate;

selecting a second neural texture of the set of neural textures based on the kernel size;

interpolating the first location of the first neural texture to a second location within the second neural texture; and

selecting a reflectance feature vector at the second location as the reflectance feature vector that represents the texel.

5. The method of claim 1 , wherein the offset-prediction neural network is trained using an unsupervised training algorithm.

6. The method of claim 1 , wherein the decoder neural network includes a multi-layer perceptron network.

7. The method of claim 1 , wherein the offset-prediction neural network and the decoder neural network were trained together using a backpropagation algorithm.

8. The method of claim 1 , wherein the decoder neural network was trained separately from a training of the offset-prediction neural network.

9. A system comprising:

a neural offset module configured to:

receive image metadata for rendering a texel of a textured surface, wherein the image metadata includes a coordinate identifying a location of the texel, an incoming radiance direction, an outgoing radiance direction, and a kernel size; and

apply an offset-prediction neural network to the coordinate and the incoming radiance direction to predict an offset coordinate of the texel;

a neural-texture query module configured to:

receive, from the neural offset module, the offset coordinate of the texel; and inputting the offset coordinate to a data structure to determine a reflectance feature vector of the texel at the kernel size, wherein: (i) the data structure includes a set of neural textures; (ii) each neural texture of the set of neural textures includes a set of reflectance feature vectors at a respective kernel size; and (iii) each reflectance feature vector of the set of reflectance feature vectors includes one or more latent variables that encode visual characteristics of a respective texel;

a decoder configured to:

identify, from the image metadata, the incoming radiance direction and the outgoing radiance direction;

receive, from the neural-texture query module, the reflectance feature vector of the texel; and

apply a decoder neural network to the reflectance feature vector, the incoming radiance direction, and the outgoing radiance direction to estimate a light- 24 reflectance value for the texel; and

a rendering engine configured to render, based on the light-reflectance value, the texel of the textured surface at the location, wherein the texel depicts at least part of the textured surface at an image scale that corresponds to the kernel size.

10. The system of claim 9 , wherein the data structure is a neural-texture pyramid, and wherein values of the one or more latent variables of a reflectance feature vector are learned using a backpropagation algorithm.

11. The system of claim 9 , wherein the neural offset module is further configured to:

querying a neural-offset texture using the coordinate to identify an offset feature vector of the coordinate;

applying the offset-prediction neural network to the offset feature vector and the incoming radiance direction to estimate a ray depth associated with the coordinate; and

applying an offset function to the ray depth to predict the offset coordinate.

12. The system of claim 9 , wherein the neural-texture query module is further configured to:

identifying, from a first neural texture of the set of neural textures, a first location within the first neural texture based on bilinear interpolation of the offset coordinate;

selecting a second neural texture of the set of neural textures based on the kernel size;

interpolating the first location of the first neural texture to a second location within the second neural texture; and

selecting a reflectance feature vector at the second location as the reflectance feature vector that represents the texel.

13. The system of claim 9 , wherein the offset-prediction neural network is trained using an unsupervised training algorithm.

14. The system of claim 9 , wherein the decoder neural network includes a multi-layer perceptron network.

15. The system of claim 9 , wherein the offset-prediction neural network and the decoder neural network were trained together using a backpropagation algorithm.

16. The system of claim 9 , wherein the decoder neural network was trained separately from a training of the offset-prediction neural network.

17. A non-transitory computer-readable storage medium including instructions configured to cause one or more data processors to perform actions including:

receiving image metadata for rendering a texel of a textured surface, wherein the image metadata includes a coordinate identifying a location of the texel, an incoming radiance direction, an outgoing radiance direction, and a kernel size;

a step for estimating a light-reflectance value for the texel by: (i) applying an offset-prediction neural network to the coordinate and the incoming radiance direction to predict an offset coordinate of the texel; (ii) inputting the offset coordinate to a data structure to determine a reflectance feature vector of the texel at the kernel size; and (iii) applying a decoder neural network to the reflectance feature vector, the incoming radiance direction, and the outgoing radiance direction to estimate a light-reflectance value for the texel; and

rendering, based on the light-reflectance value, the texel of the textured surface at the location, wherein the texel depicts at least part of the textured surface at an image scale that corresponds to the kernel size.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the data structure is a neural-texture pyramid, and wherein values of one or more latent variables of a reflectance feature vector are learned using a backpropagation algorithm.

19. The non-transitory computer-readable storage medium of claim 17 , wherein applying the offset-prediction neural network to the coordinate and the incoming radiance direction further includes:

querying a neural-offset texture using the coordinate to identify an offset feature vector of the coordinate;

applying the offset-prediction neural network to the offset feature vector and the incoming radiance direction to estimate a ray depth associated with the coordinate; and

applying an offset function to the ray depth to predict the offset coordinate.

20. The non-transitory computer-readable storage medium of claim 17 , wherein the offset-prediction neural network is trained using an unsupervised training algorithm.