Denoising Monte Carlo renderings using neural networks with asymmetric loss
A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.
1. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising:
receiving an input image rendered by MC path tracing, the input image including a plurality of pixels and input color data for each pixel;
receiving a reference image corresponding to the input image, the reference image including the plurality of pixels and reference color data for each pixel;
receiving a proportionality constant factor for each pixel, the proportionality constant factor being a positive real number;
configuring a neural network comprising:
an input layer configured to receive the input image;
a plurality of hidden layers including a plurality of nodes, a first layer of the plurality of hidden layers coupled to the input layer; and
an output layer coupled to a last layer of the plurality of hidden layers and configured to output an output image corresponding to the input image, the output image comprising the plurality of pixels and including output color data for each pixel; and
training the neural network using the input image and the reference image, the training comprising:
performing a first iteration of the training to obtain a first output image and an intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers;
determining values of a loss function for the plurality of pixels by:
for each respective pixel of the plurality of pixels:
determining a first difference between output color data and reference color data for the respective pixel;
determining a second difference between input color data and reference color data for the respective pixel;
upon determining that the first difference and the second difference have a same sign, assigning a first respective value of the loss function for the respective pixel that relates to an absolute value of the first difference multiplied by a first proportionality constant of unity; and
upon determining that the first difference and the second difference have opposite signs, assigning a second respective value of the loss function for the respective pixel that relates to the absolute value of the first difference multiplied by a second proportionality constant that is equal to unity plus the proportionality constant factor for the respective pixel;
back-propagating a gradient of the loss function through the neural network to obtain an updated intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers; and
performing additional iterations of the training to obtain a final output image and an optimized set of parameters associated with the plurality of nodes of the plurality of hidden layers.
2. The method of claim 1 , further comprising:
receiving a new input image rendered by MC path tracing; and
generating a denoised image corresponding to the new input image by passing the new input image through the neural network using the optimized set of parameters.
3. The method of claim 1 , wherein the proportionality constant factor varies from pixel to pixel.
4. The method of claim 1 , wherein the proportionality constant factor is same for all pixels.
5. The method of claim 1 , wherein the neural network comprises a convolutional neural network.
6. The method of claim 5 , wherein the neural network comprises:
a kernel prediction module configured to generate a set of weights associated with a neighborhood of pixels around each pixel; and
a reconstruction module configured to reconstruct the output image using the set of weights.
7. The method of claim 1 , wherein the neural network comprises a multilayer perceptron neural network.
8. The method of claim 1 , wherein the loss function comprises relative squared error (rSE).
9. The method of claim 1 , wherein the loss function comprises symmetric absolute percentage error (SAPE).
10. The method of claim 1 , wherein the loss function comprises structural similarity (SSIM) error or multi-scale structural similarity (MS-SSIM) error.
11. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that when executed control a computer system to denoise images rendered by Monte Carlo (MC) path tracing, the instructions comprising:
receiving an input image rendered by MC path tracing, the input image including a plurality of pixels and input color data for each pixel;
receiving a reference image corresponding to the input image, the reference image including the plurality of pixels and reference color data for each pixel;
receiving a proportionality constant factor for each pixel, the proportionality constant factor being a positive real number;
configuring a neural network comprising:
an input layer configured to receive the input image;
a plurality of hidden layers including a plurality of nodes, a first layer of the plurality of hidden layers coupled to the input layer; and
an output layer coupled to a last layer of the plurality of hidden layers and configured to output an output image corresponding to the input image, the output image comprising the plurality of pixels and including output color data for each pixel; and
training the neural network using the input image and the reference image, the training comprising:
performing a first iteration of the training to obtain a first output image and an intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers;
determining values of a loss function for the plurality of pixels by:
for each respective pixel of the plurality of pixels:
determining a first difference between output color data and reference color data for the respective pixel;
determining a second difference between input color data and reference color data for the respective pixel;
upon determining that the first difference and the second difference have a same sign, assigning a first respective value of the loss function for the respective pixel that relates to an absolute value of the first difference multiplied by a first proportionality constant of unity; and
upon determining that the first difference and the second difference have opposite signs, assigning a second respective value of the loss function for the respective pixel that relates to the absolute value of the first difference multiplied by a second proportionality constant that is equal to unity plus the proportionality constant factor for the respective pixel;
back-propagating a gradient of the loss function through the neural network to obtain an updated intermediate set of parameters associated with the plurality of nodes of the plurality of hidden layers; and
performing additional iterations of the training to obtain a final output image and an optimized set of parameters associated with the plurality of nodes of the plurality of hidden layers.
12. The computer product of claim 11 , wherein the instructions further comprising:
receiving a new input image rendered by MC path tracing; and
generating a denoised image corresponding to the new input image by passing the new input image through the neural network using the optimized set of parameters.
13. The computer product of claim 11 , wherein the proportionality constant factor varies from pixel to pixel.
14. The computer product of claim 11 , wherein the proportionality constant factor is same for all pixels.
15. The computer product of claim 11 , wherein the neural network comprises a convolutional neural network.
16. The computer product of claim 15 , wherein the neural network comprises
a kernel prediction module configured to generate a set of weights associated with a neighborhood of pixels around each pixel; and
a reconstruction module configured to reconstruct the output image using the set of weights.
17. The computer product of claim 11 , wherein the neural network comprises a multilayer perceptron neural network.
18. The computer product of claim 11 , wherein the loss function comprises relative squared error (rSE).
19. The computer product of claim 11 , wherein the loss function comprises symmetric absolute percentage error (SAPE).
20. The computer product of claim 11 , wherein the loss function comprises structural similarity (SSIM) error or multi-scale structural similarity (MS-S SIM) error.