Real-time neural appearance models
Embodiments of the present disclosure relate to real-time neural appearance models. Using a neural decoder, scenes are rendered in real-time with complex material appearance previously reserved for offline use. Learned hierarchical textures representing the material properties are encoded as latent codes. When a ray is cast and intersects with geometry in the scene, the intersection point is mapped to one of the latent codes. The latent code is interpreted using neural decoders, which produce reflectance values and importance-sampled directions that can be used to determine a pixel color.
1 . A computer-implemented method, comprising:
receiving a latent code defining properties of a material associated with a surface at a point intersected by a ray;
extracting, from the latent code, at least one property comprising a learned shading frame defining a local orientation of the material;
transforming a surface incident vector and a surface outgoing vector at the point using the at least one extracted property to produce transformed properties; and
predicting, by an attribute decoder neural network, reflectance attributes for the surface based on the latent code and the transformed properties.
2 . The computer-implemented method of claim 1 , wherein the material comprises at least one layer including one or more of a low-roughness dielectric coating, conductor with glints, or stains.
3 . The computer-implemented method of claim 1 , further comprising processing the surface incident vector, normal and tangent vectors at the point intersected by the ray, and the latent code by a neural sampling decoder to predict a next segment of a path that includes the ray.
4 . The computer-implemented method of claim 1 , further comprising predicting directional albedo, directionless albedo, roughness estimation, or transmissivity for the surface based on the latent code and the transformed properties.
5 . The computer-implemented method of claim 1 , further comprising obtaining the latent code from a stored texture using coordinates of the point.
6 . The computer-implemented method of claim 1 , further comprising processing, by a material encoder, at least one of coordinates and surface parameters of the point intersected by the ray according to learned parameters to generate the latent code.
7 . The computer-implemented method of claim 1 , wherein the properties of the material comprise albedo, roughness, a normal vector, and a tangent vector.
8 . The computer-implemented method of claim 1 , wherein the material comprises at least one volumetric layer.
9 . The computer-implemented method of claim 1 , wherein matrix operations implemented by the attribute decoder neural network are executed by a tensor core.
10 . The computer-implemented method of claim 1 , wherein at least one of the steps of receiving, extracting, transforming, or predicting is performed on a server or in a data center to generate an image, and the image is streamed to a user device.
11 . The computer-implemented method of claim 1 , wherein at least one of the steps of receiving, extracting, transforming, or predicting is performed within a cloud computing environment.
12 . The computer-implemented method of claim 1 , wherein at least one of the steps of receiving, extracting, transforming, or predicting is performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.
13 . The computer-implemented method of claim 1 , wherein at least one of the steps of receiving, extracting, transforming, or predicting is performed on a virtual machine comprising a portion of a graphics processing unit.
14 . The computer-implemented method of claim 1 , wherein the transforming comprises:
constructing a combined transformation matrix based on the learned shading frame; and
computing the transformed properties as a first product of the surface incident vector and the combined transformation matrix and a second product of the surface outgoing vector and the combined transformation matrix.
15 . A computer-implemented method, comprising:
processing, by a material encoder, at least one of coordinates and surface parameters of a point intersected by a ray according to learned parameters to generate a latent code defining properties of a material associated with the surface at the point;
extracting, from the latent code, at least one of the properties of the material;
transforming a surface incident vector and a surface outgoing vector at the point using the at least one extracted property to produce transformed properties; and
predicting, by an attribute decoder neural network, reflectance attributes for the surface based on the latent code and the transformed properties, wherein the material encoder is trained by:
determining sample points on the surface;
at each sample point of the sample points, processing the properties of the material, by the material encoder, according to learned parameters to generate estimated latent codes; and
processing the estimated latent codes to predict estimated reflectance attributes at the sample points.
16 . The computer-implemented method of claim 15 , further comprising adjusting the learned parameters based on differences between the estimated reflectance attributes and reference reflectance attributes.
17 . The computer-implemented method of claim 15 , further comprising adjusting, based on differences between the estimated reflectance attributes and reference reflectance attributes, decoding parameters that are applied to the estimated latent codes by the attribute decoder neural network to predict the estimated reflectance attributes.
18 . A system, comprising:
a memory that stores a surface; and
a processor that is connected to the memory, wherein the processor is configured to predict reflectance attributes for the surface by:
receiving a latent code defining properties of a material associated with the surface at a point intersected by a ray;
extracting, from the latent code, at least one property comprising a learned shading frame defining a local orientation of the material;
transforming a surface incident vector and a surface outgoing vector at the point using the at least one extracted property to produce transformed properties; and
predicting, by an attribute decoder neural network, the reflectance attributes for the surface based on the latent code and the transformed properties.
19 . The system of claim 18 , further comprising obtaining the latent code from a stored texture using coordinates of the point.
20 . A non-transitory computer-readable medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving a latent code defining properties of a material associated with a surface at a point intersected by a ray;
extracting, from the latent code, at least one property comprising a learned shading frame defining a local orientation of the material;
transforming a surface incident vector and a surface outgoing vector at the point using the at least one extracted property to produce transformed properties; and
predicting, by an attribute decoder neural network, reflectance attributes for the surface based on the latent code and the transformed properties.
21 . The non-transitory computer-readable medium of claim 20 , further comprising processing the surface incident vector, normal and tangent vectors at the point intersected by the ray, and the latent code by a neural sampling decoder to predict a next segment of a path that includes the ray.