Temporally amortized supersampling using a kernel splatting network
One embodiment provides a graphics processor comprising processing resources configured to perform a supersampling anti-aliasing operation via a mixed precision convolutional neural network. The processing resources include circuitry configured to receive, at an input block of a neural network model, a data including previous frame data, current frame data, jitter offset data, and velocity data, pre-process the data to generate pre-processed data, provide pre-processed data to a feature extraction network of the neural network model and an output block of the neural network model, process the first pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages, output tensor data from the feature extraction network to the output block, and generate an anti-aliased output frame via the output block based on the current frame data and the tensor data output from the feature extraction network.
1 . A graphics processor comprising:
a set of processing resources configured to perform a supersampling anti-aliasing operation via a mixed precision convolutional neural network, the set of processing resources including circuitry configured to:
receive, at an input block of a neural network model, a set of data including previous frame data, current frame data, jitter offset data, and velocity data;
pre-process the set of data to generate pre-processed data;
provide the pre-processed data to a feature extraction network of the neural network model;
process the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
output tensor data from the feature extraction network to an output block of the neural network model; and
generate an output frame via the output block based on the current frame data and the tensor data via kernel splatting and blending operations performed on the current frame data and the previous frame data.
2 . The graphics processor as in claim 1 , wherein the previous frame data includes one or more previously generated output frames, the current frame data includes output of a raster and lighting stage of a render pipeline, the velocity data includes motion vectors generated by the render pipeline, and the jitter offset data indicates a two dimensional camera offset applied to the current frame data.
3 . The graphics processor as in claim 2 , wherein the tensor data includes a first set of weights and a second set of weights.
4 . The graphics processor as in claim 3 , wherein to generate the output frame includes to:
upsample the current frame data via the first set of weights to generate an upscaled image at a target resolution; and
blend the upscaled image with the previous frame data via the second set of weights.
5 . The graphics processor as in claim 4 , wherein to upsample the current frame data via the first set of weights includes to generate an upsampling kernel based on the first set of weights and apply the upsampling kernel to the current frame data to splat respective samples of the current frame to surrounding pixels.
6 . The graphics processor as in claim 5 , wherein the upsampling kernel is a 3×3×4 per-sample kernel and the first set of weights include spatial weights to indicate a contribution of a sample to surrounding pixels.
7 . The graphics processor as in claim 5 , wherein the pre-processed data is provided at an integer precision and the output block is to generate the output frame via floating-point operations.
8 . The graphics processor as in claim 5 , wherein to pre-process the set of data includes to:
warp the previous frame data based on the velocity data to generate warped history data; and
shuffle the warped history data and current frame data from a spatial dimension to a channel dimension.
9 . The graphics processor as in claim 8 , wherein the channel dimension includes a depth channel.
10 . A method comprising:
receiving, at an input block of a neural network model, a set of data including previous frame data, current frame data, jitter offset data, and velocity data, wherein the neural network model is a mixed precision neural network model;
pre-processing the set of data to generate pre-processed data;
providing pre-processed data to a feature extraction network of the neural network model;
processing the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
outputting tensor data from the feature extraction network to an output block of the neural network model; and
generating an output frame via the output block based on the current frame data and the tensor data via kernel splatting and blending operations performed on the current frame data and the previous frame data.
11 . The method as in claim 10 , wherein the previous frame data includes one or more previously generated output frames, the current frame data includes output of a raster and lighting stage of a render pipeline, the velocity data includes motion vectors generated by the render pipeline, and the jitter offset data indicates a two dimensional camera offset applied to the current frame data.
12 . The method as in claim 11 , wherein the tensor data includes a first set of weights and a second set of weights.
13 . The method as in claim 12 , wherein generating the output frame includes:
upsampling the current frame data via the first set of weights to generate an upscaled image at a target resolution; and
blending the upscaled image with the previous frame data via the second set of weights.
14 . The method as in claim 13 , wherein upsampling the current frame data via the first set of weights includes generating an upsampling kernel based on the first set of weights and applying the upsampling kernel to the current frame data to splat respective samples of the current frame data to surrounding pixels.
15 . The method as in claim 14 , wherein the upsampling kernel is a 3×3×4 per-sample kernel and the first set of weights include spatial weights to indicate a contribution of a sample to surrounding pixels.
16 . The method as in claim 14 , additionally comprising generating an output frame at the output block via floating-point operations, wherein the pre-processed data is provided at an integer precision.
17 . The method as in claim 14 , wherein pre-processing the set of data includes:
warping the previous frame data based on the velocity data to generate warped history data; and
shuffling the warped history data and current frame data from a spatial dimension to a channel dimension.
18 . The method as in claim 17 , wherein the channel dimension includes a depth channel.
19 . A non-transitory machine-readable medium storing instructions which, when executed by one or more processors cause the one or more processors to perform operations comprising:
receiving, at an input block of a neural network model, a set of data including previous frame data, current frame data, jitter offset data, and velocity data, wherein the neural network model is a mixed precision neural network model;
pre-processing the set of data to generate pre-processed data;
providing pre-processed data to a feature extraction network of the neural network model;
processing the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
outputting tensor data from the feature extraction network to an output block of the neural network model; and
generating an output frame via the output block based on the current frame data and the tensor data via kernel splatting and blending operations performed on the current frame data and the previous frame data, including splatting respective samples of the current frame data to surrounding pixels via a per-sample upsampling kernel configured to a contribution of a sample it surrounding pixels.
20 . The non-transitory machine-readable medium as in claim 19 , wherein the previous frame data includes one or more previously generated output frames, the current frame data includes output of a raster and lighting stage of a render pipeline, the velocity data includes motion vectors generated by the render pipeline, and the jitter offset data indicates a two dimensional camera offset applied to the current frame data.