Temporally amortized supersampling using a mixed precision convolutional neural network
One embodiment provides a graphics processor comprising a set of processing resources configured to perform a supersampling operation via a mixed precision convolutional neural network, the set of processing resources including circuitry configured to receive, at an input block of a neural network model, history data, velocity data, and current frame data, pre-process the history data, velocity data, and current frame data to generate pre-processed data, provide the pre-processed data to a feature extraction network of the neural network model, process the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages, and generate an output image via an output block of the neural network model via direct reconstruction or kernel prediction.
1. A graphics processor comprising:
a set of processing resources configured to perform a supersampling anti-aliasing operation via a mixed precision convolutional neural network, the set of processing resources including circuitry configured to:
receive, at an input block of a neural network model, a set of data including previous frame data, current frame data, and velocity data, wherein the previous frame data includes one or more previously generated output frames, the current frame data includes output of a raster and lighting stage of a render pipeline, and the velocity data includes motion vectors generated by the render pipeline;
pre-process the set of data to generate pre-processed data;
provide first pre-processed data to a feature extraction network of the neural network model and second pre-processed data to an output block of the neural network model, the first pre-processed data in a datatype having a first precision and the second pre-processed data in a datatype having a second precision that is higher than the first precision;
process the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
output tensor data from the feature extraction network to the output block; and
generate an output frame via the output block based on the second pre-processed data from the input block and the tensor data output from the feature extraction network, wherein the output frame is an anti-aliased frame.
2. The graphics processor as in claim 1 , wherein to generate the pre-processed data includes to:
warp the previous frame data based on the velocity data to generate warped history data;
align the warped history data with the current frame data to generate aligned history data; and
shuffle the aligned history data and current frame data from a spatial dimension to a channel dimension.
3. The graphics processor as in claim 2 , wherein the channel dimension includes a depth channel or multiple feature map channels.
4. The graphics processor as in claim 2 , wherein to generate an output frame via the output block includes to generate the output frame via one or more convolution layers of the output block.
5. The graphics processor as in claim 2 , wherein to generate an output frame via the output block includes to predict a set of per-pixel kernel values and blend weights, filter the current frame data via the per-pixel kernel values, and blend the aligned history data with filtered current frame data.
6. The graphics processor as in claim 2 , wherein the circuitry is configured to upscale the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution before the current frame data is provided to the input block of the neural network model.
7. The graphics processor as in claim 2 , wherein to generate the pre-processed data includes to upscale the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution.
8. The graphics processor as in claim 2 , wherein to generate the pre-processed data additionally includes to apply a convolution and an activation function to shuffled aligned history data and current frame data.
9. A method comprising:
receiving, at an input block of a neural network model, a set of data including previous frame data, current frame data, and velocity data, wherein the previous frame data includes one or more previously generated output frames, the current frame data includes output of a raster and lighting stage of a render pipeline, and the velocity data includes motion vectors generated by the render pipeline;
pre-processing the set of data to generate pre-processed data;
providing first pre-processed data to a feature extraction network of the neural network model and second pre-processed data to an output block of the neural network model, the first pre-processed data in a datatype having a first precision and the second pre-processed data in a datatype having a second precision that is higher than the first precision;
processing the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
outputting tensor data from the feature extraction network to the output block; and
generating an output frame via the output block based on the second pre-processed data from the input block and the tensor data output from the feature extraction network, wherein the output frame is an anti-aliased frame.
10. The method as in claim 9 , wherein generating the pre-processed data includes:
warping the previous frame data based on the velocity data to generate warped history data;
aligning the warped history data with the current frame data to generate aligned history data; and
shuffling the aligned history data and current frame data from a spatial dimension to a channel dimension.
11. The method as in claim 10 , wherein the channel dimension includes a depth channel or multiple feature map channels and generating pre-processed data additionally includes applying a convolution and an activation function to shuffled aligned history data and current frame data.
12. The method as in claim 10 , wherein generating an output frame via the output block includes generating the output frame via one or more convolution layers of the output block.
13. The method as in claim 10 , wherein generating an output frame via the output block includes to predict a set of per-pixel kernel values and blend weights, filter the current frame data via the per-pixel kernel values, and blend the aligned history data with filtered current frame data.
14. The method as in claim 10 , further comprising upscaling the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution before the current frame data is provided to the input block of the neural network model.
15. The method as in claim 10 , wherein generating the pre-processed data includes upscaling the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution.
16. A data processing system comprising:
a memory device; and
a graphics processor coupled with the memory device, the graphics processor comprising a set of processing resources configured to perform a supersampling anti-aliasing operation via a mixed precision convolutional neural network, the set of processing resources including circuitry configured to:
receive, at an input block of a neural network model, a set of data including previous frame data, current frame data, and velocity data, wherein the previous frame data includes one or more previously generated output frames, the current frame data includes output of a raster and lighting stage of a render pipeline, and the velocity data includes motion vectors generated by the render pipeline;
pre-process the set of data to generate pre-processed data;
provide first pre-processed data to a feature extraction network of the neural network model and second pre-processed data to an output block of the neural network model, the first pre-processed data in a datatype having a first precision and the second pre-processed data in a datatype having a second precision that is higher than the first precision;
process the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
output tensor data from the feature extraction network to the output block; and
generate an output frame via the output block based on the second pre-processed data from the input block and the tensor data output from the feature extraction network, wherein the output frame is an anti-aliased frame.
17. The data processing system as in claim 16 , wherein to generate the pre-processed data includes to:
upscale the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution;
warp the previous frame data based on the velocity data to generate warped history data;
align the warped history data with the current frame data to generate aligned history data; and
shuffle the aligned history data and current frame data from a spatial dimension to a channel dimension.
18. The data processing system as in claim 17 , wherein the channel dimension includes a depth channel or multiple feature map channels.
19. The data processing system as in claim 17 , wherein to generate the output frame via the output block includes to generate the output frame via one or more convolution layers of the output block.
20. The data processing system as in claim 17 , wherein to generate the output frame via the output block includes to predict a set of per-pixel kernel values and blend weights, filter the current frame data via the per-pixel kernel values, and blend the aligned history data with filtered current frame data.