Temporal techniques of denoising Monte Carlo renderings using neural networks
A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.
1. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising:
receiving a sequence of frames rendered by MC path tracing, the sequence of frames including a center frame and one or more temporal neighboring frames;
receiving a reference image corresponding to the center frame;
configuring a plurality of first neural networks, each respective first neural network configured to extract a respective set of first features from a respective frame of the sequence of frames;
configuring a second neural network including a plurality of layers and a plurality of nodes, the second neural network configured to:
extract a set of temporal features from the sets of first features; and
output an output frame corresponding to the center frame; and
training the second neural network to obtain a plurality of optimized parameters associated with the plurality of nodes of the second neural network using the sequence of frames and the reference image corresponding to the center frame.
2. The method of claim 1 , further comprising:
receiving a new sequence of frames rendered by MC path tracing, the new sequence of frames including a new center frame; and
generating a denoised frame corresponding to the new center frame by passing the new sequence of frames through the plurality of first neural networks and the second neural network using the plurality of optimized parameters associated with the plurality of nodes of the second neural network.
3. The method of claim 1 , wherein each respective first neural network of the plurality of first neural networks includes a respective plurality of layers and a respective plurality of nodes, and each respective first neural network is pre-trained to obtain a respective plurality of optimized parameters associated the respective plurality of nodes of the respective first neural network.
4. The method of claim 3 , wherein the plurality of first neural networks share a same plurality of optimized parameters.
5. The method of claim 3 , wherein each respective first neural network has a respective plurality of optimized parameters independent from other first neural networks of the plurality of first neural networks.
6. The method of claim 1 , wherein each respective first neural network of the plurality of first neural networks includes a respective plurality of layers and a respective plurality of nodes, and each respective first neural network is jointly-trained with the training of the second neural network to obtain a respective plurality of optimized parameters associated with the respective plurality of nodes of the respective first neural network.
7. The method of claim 1 , wherein each respective first neural network comprises a spatial-feature extractor, and the respective set of first features comprises a respective set of spatial features.
8. The method of claim 1 , wherein the one or more temporal neighboring frames in the sequence of frames include one or more past frames and one or more future frames relative to the center frame, or one or more past frames only, or one or more future frames only.
9. The method of claim 1 , wherein each first neural network comprises a convolutional neural network, and the second neural network comprises a convolutional neural network.
10. The method of claim 9 , wherein the second neural network comprises a plurality of residual blocks.
11. The method of claim 9 , wherein the second neural network comprises:
a kernel prediction module configured to generate a respective set of weights for each respective frame of the sequence of frames, the respective set of weights associated with a neighborhood of pixels around each pixel of the respective frame; and
a reconstruction module configured to reconstruct a plurality of denoised frames, each denoised frame corresponding to a respective frame of the sequence of frames and reconstructed using a respective set of weights.
12. The method of claim 11 , wherein the sets of weights are jointly normalized.
13. The method of claim 11 , further comprising motion-warping each respective set of first features to the center frame using a respective motion vector, and wherein the second neural network is configured to extract the set of temporal features from the sets of first features that have been motion-warped.
14. The method of claim 13 , wherein the reconstruction module is configured to reconstruct each respective denoised frame by:
offsetting the respective set of weights along an inverted motion-vector path back to a corresponding frame to obtain respective set of offset weight; and
applying the respective set of offset weights to the corresponding frame to obtain the respective denoised frame.
15. The method of claim 1 , wherein each first neural network comprises a multilayer perceptron neural network, and the second neural network comprises a multilayer perceptron neural network.
16. A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising:
receiving a sequence of frames rendered by MC path tracing, the sequence of frames including a center frame and one or more temporal neighboring frames;
receiving a reference image corresponding to the center frame;
configuring a plurality of first neural networks, each respective first neural network comprising a respective first plurality of layers and a respective first set of nodes, each respective first neural network configured to extract a respective set of first features from a respective frame of the sequence of frames;
configuring a second neural network including a second plurality of layers and a second set of nodes, the second neural network configured to:
extract a set of temporal features from the sets of first features; and
output an output frame corresponding to the center frame; and
training the plurality of first neural networks and the second neural network to obtain a respective first set of optimized parameters associated with each respective first set of nodes of the respective first neural network, and a second set of optimized parameters associated with the second set of nodes of the second neural network, wherein the training uses the sequence of frames and the reference image corresponding to the center frame.
17. The method of claim 16 , further comprising:
receiving a new sequence of frames rendered by MC path tracing, the new sequence of frames including a new center frame; and
generating a denoised frame corresponding to the new center frame by passing the new sequence of frames through the plurality of first neural networks and the second neural network using the first sets of optimized parameters associated with the plurality of first neural networks and the second set of optimized parameters associated with the second neural network.
18. The method of claim 16 , wherein each respective first neural network comprises a respective source encoder configured to extract a set of low-level features from a respective frame.
19. The method of claim 18 , wherein each respective first neural network further comprises a respective spatial-feature extractor configured to receive the set of low-level features extracted by the respective source encoder, and to extract the respective set of first features from the respective frame.
20. The method of claim 16 , wherein the second neural network comprises:
a kernel prediction module configured to generate a respective set of weights for each respective frame of the sequence of frames, wherein the respective set of weights is associated with a neighborhood of pixels around each pixel of the respective frame; and
a reconstruction module configured to:
reconstruct a plurality of denoised frames, each denoised frame is corresponding to a respective frame of the sequence of frames and is reconstructed using a respective set of weights; and
reconstruct the output frame by combining the plurality of denoised frames.