Sample distribution-informed denoising and rendering
A graphics processor is provided that includes circuitry configured to receive, at an input block of a neural network model, a set of data including previous frame data, current frame data, velocity data, and jitter offset data. The neural network model is configured to generate a denoised, supersampled, and anti-aliased output image based on reliability metrics computed based on sample distribution data for samples within the current frame data.
1 . A graphics processor comprising:
a set of processing resources configured to perform a supersampling anti-aliasing operation via a mixed precision convolutional neural network, the set of processing resources including circuitry configured to:
receive, at an input block of a neural network model, a set of data including previous frame data, current frame data, velocity data, and jitter offset data;
pre-process the set of data to generate pre-processed data;
provide first pre-processed data to a feature extraction network of the neural network model and second pre-processed data to an output block of the neural network model, the first pre-processed data in a datatype having a first precision and the second pre-processed data in a datatype having a second precision that is higher than the first precision;
process the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
output tensor data from the feature extraction network to the output block; and
generate an output image via an output block of the neural network model, wherein the output image is a denoised, supersampled, and anti-aliased output image and the output block is configured to filter the output image based on sample reliability metrics computed based on sample distribution data for samples within the current frame data.
2 . The graphics processor as in claim 1 , wherein the circuitry is configured to receive, at the input block, the sample reliability metrics and to filter the output image, the circuitry is configured to denoise the output image based on the sample reliability metrics.
3 . The graphics processor as in claim 2 , wherein the current frame data includes high dynamic range (HDR) color data.
4 . The graphics processor as in claim 3 , wherein the current frame data includes G-buffer data including auxiliary image data.
5 . The graphics processor as in claim 4 , wherein the auxiliary image data includes albedo and normal data and the sample reliability metrics are configured at least in part to compensate for noise within the auxiliary image data.
6 . The graphics processor as in claim 5 , wherein the sample reliability metrics include a moment of a statistical distribution of sample characteristics for samples within the current frame data.
7 . The graphics processor as in claim 6 , wherein the sample reliability metrics are computed based on a perceptually uniform encoding of the samples in which distortion visibility is approximately uniform along encoded values.
8 . The graphics processor as in claim 7 , wherein variance of the sample distribution data is characterized by a Jensen gap of perceptually uniform, tone-mapped samples.
9 . The graphics processor as in claim 8 , wherein the circuitry includes a matrix accelerator that is configured to perform matrix operations for the neural network model.
10 . The graphics processor as in claim 9 , wherein the matrix accelerator includes a systolic array.
11 . A method comprising:
on a graphics processor device:
receiving, at an input block of a neural network model, a set of data including previous frame data, current frame data, velocity data, and jitter offset data;
pre-processing the set of data to generate pre-processed data;
providing first pre-processed data to a feature extraction network of the neural network model and second pre-processed data to an output block of the neural network model, the first pre-processed data in a datatype having a first precision and the second pre-processed data in a datatype having a second precision that is higher than the first precision;
processing the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
outputting tensor data from the feature extraction network to the output block; and
generating an output image via an output block of the neural network model, wherein the output image is a denoised, supersampled, and anti-aliased output image and the output block is configured to filter the output image based on sample reliability metrics computed based on sample distribution data for samples within the current frame data.
12 . The method as in claim 11 , further comprising receiving, at the input block, the sample reliability metrics for the current frame data, wherein filtering the output image includes denoising the output image based on the sample reliability metrics, wherein, and the sample reliability metrics include a moment of a statistical distribution of sample characteristics for samples within the current frame data.
13 . The method as in claim 12 , wherein the current frame data includes high dynamic range (HDR) color data.
14 . The method as in claim 13 , wherein the current frame data includes G-buffer data including auxiliary image data.
15 . The method as in claim 14 , wherein the auxiliary image data includes albedo and normal data and the sample reliability metrics are configured at least in part to compensate for noise within the auxiliary image data.
16 . A non-transitory machine readable medium storing instructions which, when executed by one or more processors including a graphics processor, cause the one or more processors to perform operations comprising:
receiving, at an input block of a neural network model, a set of data including previous frame data, current frame data, velocity data, jitter offset data, and sample reliability metrics determined based on sample distribution data for samples within the current frame data;
pre-processing the set of data to generate pre-processed data;
providing first pre-processed data to a feature extraction network of the neural network model and second pre-processed data to an output block of the neural network model, the first pre-processed data in a datatype having a first precision and the second pre-processed data in a datatype having a second precision that is higher than the first precision;
processing the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
outputting tensor data from the feature extraction network to the output block; and
generating an output image via an output block of the neural network model, wherein the output image is a denoised, supersampled, and anti-aliased output image and the output block is configured to filter the output image based on the sample reliability metrics.
17 . The non-transitory machine readable medium as in claim 16 , wherein filtering the output image includes denoising the output image based on sample reliability metrics, wherein the sample reliability metrics include a moment of a statistical distribution of sample characteristics for samples within the current frame data.
18 . The non-transitory machine readable medium as in claim 17 , wherein the current frame data includes high dynamic range (HDR) color data.
19 . The non-transitory machine readable medium as in claim 18 , wherein the current frame data includes G-buffer data including auxiliary image data.
20 . The non-transitory machine readable medium as in claim 19 , wherein the auxiliary image data includes albedo and normal data and the sample reliability metrics are configured at least in part to compensate for noise within the auxiliary image data.