Data-driven modeling of advanced paint appearance

1. A computer-implemented method for rendering a painting stroke of a three-dimensional digital painting, the method comprising:

receiving a painting stroke input on a canvas, wherein the painting stroke comprises a plurality of pixels;

for each pixel in the plurality of pixels:

selecting a neighborhood patch of pixels,

inputting the neighborhood patch of pixels into a neural network, and

outputting a shading function from the neural network in real-time after receiving the painting stroke, wherein the shading function represents a global light transport effect of the pixel across the neighborhood patch of pixels and wherein the neural network outputs the shading function as a four dimensional (4D) tensor by:

using encoding layers of the neural network to reduce a resolution of the neighborhood patch of pixels and increase a feature map of the neighborhood patch of pixels;

reshaping a final encoding layer to a 4D table; and

using decoding layers of the neural network to increase the resolution of the 4D table and decrease a number of features of the 4D table to output the 4D tensor representation of the shading function comprising a color feature; and

rendering the painting stroke on the canvas using the shading function.

2. The method of claim 1 , wherein the neighborhood patch of pixels comprises an area defined by a geometric shape with the pixel as a center of the geometric shape and the geometric shape aligned on the pixel.

3. The method of claim 2 , wherein the geometric shape comprises a square.

4. The method of claim 1 , wherein the neighborhood patch of pixels comprises a height field and materials information, the materials information including any of surface reflectivity, layered subsurface properties or pigment concentration.

5. The method of claim 1 , wherein rendering the painting stroke on the canvas comprises rendering the painting stroke on the canvas using the shading function in real time.

6. The method of claim 1 , wherein:

at least one pixel of the plurality of pixels of the painting stroke comprises the shading function for the at least one pixel;

for the at least one pixel:

selecting a neighborhood patch of pixels,

inputting the neighborhood patch of pixels into the neural network, and

outputting an updated shading function from the neural network; and

rendering the painting stroke comprises rendering the painting stroke on the canvas using the updated shading function for the at least one pixel.

7. The method of claim 1 , wherein the neural network outputs the shading function as a Gaussian mixture model (GMM) by:

using encoding layers of the neural network to reduce a resolution of the neighborhood patch of pixels and increase a feature map of the neighborhood patch of pixels; and

generating parameters for the GMM from a final encoding layer to output the parameters of the GMM representation of the shading function comprising a color feature.

8. A system for rendering a painting stroke of a three-dimensional digital painting, the system comprising:

at least one memory including instructions; and

at least one processor that is operably coupled to the at least one memory and that is arranged and configured to execute instructions that, when executed, cause the at least one processor to implement an application, the application comprising:

a painting engine that is configured to:

receive a painting stroke input on a canvas, wherein the painting stroke comprises a plurality of pixels, and

for each of the plurality of pixels, select a neighborhood patch of pixels; and

a neural network that for each pixel in the plurality of pixels is configured to:

receive the neighborhood patch of pixels, and

output a shading function, wherein the shading function comprises a four dimensional (4D) tensor that represents a global light transport effect of the pixel across the neighborhood patch of pixels,

wherein the painting engine is configured to render the painting stroke on the canvas using the shading function and wherein the neural network outputs the shading function by:

using encoding layers of the neural network to reduce a resolution of the neighborhood patch of pixels and increase a feature map of the neighborhood patch of pixels;

reshaping a final encoding layer to a 4D table; and

using decoding layers of the neural network to increase the resolution of the 4D table and decrease a number of features of the 4D table to output the 4D tensor representation of the shading function comprising a color feature.

9. The system of claim 8 , wherein the neighborhood patch of pixels comprises an area defined by a geometric shape with the pixel as a center of the geometric shape and the geometric shape aligned on the pixel.

10. The system of claim 9 , wherein the geometric shape comprises a square.

11. The system of claim 8 , wherein the neighborhood patch of pixels comprises a height field and materials information, the materials information including any of surface reflectivity, layered subsurface properties or pigment concentration.

12. The system of claim 8 , wherein the painting engine renders the painting stroke on the canvas using the shading function in real time.

13. The system of claim 8 , wherein:

at least one pixel of the plurality of pixels of the painting stroke comprises the shading function for the at least one pixel;

the neural network for the at least one pixel:

receives the neighborhood patch of pixels into the neural network, and

outputs an updated shading function from the neural network; and

the painting engine renders the painting stroke the painting stroke on the canvas using the updated shading function for the at least one pixel.

14. The system of claim 8 , wherein the neural network outputs the shading function as a Gaussian mixture model (GMM) by:

using encoding layers of the neural network to reduce a resolution of the neighborhood patch of pixels and increase a feature map of the neighborhood patch of pixels; and

generating parameters for the GMM from a final encoding layer to output the parameters of the GMM representation of the shading function comprising a color feature.

15. A computer program product for rendering a painting stroke of a three-dimensional digital painting, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to:

receive a painting stroke input on a canvas, wherein the painting stroke comprises a plurality of pixels;

for each pixel in the plurality of pixels:

selecting a neighborhood patch of pixels,

inputting the neighborhood patch of pixels into a neural network, and

outputting a shading function from the neural network, wherein the shading function takes an incoming light direction and an outgoing light direction as input and represents a global light transport effect of the pixel across the neighborhood patch of pixels, and wherein the neural network outputs the shading function as a four dimensional (4D) tensor by:

using encoding layers of the neural network to reduce a resolution of the neighborhood patch of pixels and increase a feature map of the neighborhood patch of pixels;

reshaping a final encoding layer to a 4D table; and

using decoding layers of the neural network to increase the resolution of the 4D table and decrease a number of features of the 4D table to output the 4D tensor representation of the shading function comprising a color feature; and

rendering the painting stroke on the canvas using the shading function.

16. The computer program product of claim 15 , wherein the neighborhood patch of pixels comprises an area defined by a geometric shape with the pixel as a center of the geometric shape and the geometric shape aligned on the pixel.

17. The computer program product of claim 15 , wherein the neural network outputs the shading function as a Gaussian mixture model (GMM) by:

using encoding layers of the neural network to reduce a resolution of the neighborhood patch of pixels and increase a feature map of the neighborhood patch of pixels; and

generating parameters for the GMM from a final encoding layer to output the parameters of the GMM representation of the shading function comprising a color feature.