Motion vector refinement for temporally amortized supersampling
A residual network is used to predict a set of residual motion vectors that provide additional motion data for portions of the frame for which motion vectors are not provided, such as animated textures, mirrored/reflected objects, and/or moving objects without motion information.
1. A graphics processor comprising:
a set of processing resources configured to perform a supersampling anti-aliasing operation via a mixed precision convolutional neural network, the set of processing resources including circuitry configured to:
receive, at an input block of a first neural network model, a set of data including previous frame data, current frame data, and velocity data, wherein the previous frame data includes one or more previously generated output frames, the current frame data includes output of a raster and lighting stage of a render pipeline, and the velocity data includes motion vectors generated by the render pipeline;
pre-process the set of data to generate pre-processed data, wherein to pre-process the set of data includes to generate residual motion vectors via a second neural network model;
provide first pre-processed data to a feature extraction network of the first neural network model and second-processed data to an output block of the first neural network model, the first pre-processed data at a first precision and the second pre-processed data at a second precision that is higher than the first precision;
process the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
output tensor data from the feature extraction network to the output block; and
generate an output frame via the output block based on the second pre-processed data from the input block and the tensor data output from the feature extraction network, wherein the output frame is an anti-aliased frame.
2. The graphics processor as in claim 1 , wherein to generate the pre-processed data includes to:
generate the residual motion vectors via the second neural network based on the current frame data, previous frame data, and the motion vectors;
combine the motion vectors and the residual motion vectors to create refined motion vectors;
warp the previous frame data based on the refined motion vectors to generate warped history data;
align the warped history data with the current frame data to generate aligned history data; and
shuffle the aligned history data and current frame data from a spatial dimension to a channel dimension.
3. The graphics processor as in claim 2 , wherein the channel dimension includes a depth channel or multiple feature map channels.
4. The graphics processor as in claim 2 , wherein to generate an output frame via the output block includes to generate the output frame via one or more convolution layers of the output block.
5. The graphics processor as in claim 2 , wherein to generate an output frame via the output block includes to predict a set of per-pixel kernel values and blend weights, filter the current frame data via the per-pixel kernel values, and blend the aligned history data with filtered current frame data.
6. The graphics processor as in claim 2 , wherein the circuitry is configured to upscale the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution before the current frame data is provided to the input block of the first neural network model.
7. The graphics processor as in claim 2 , wherein to generate the pre-processed data includes to upscale the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution.
8. The graphics processor as in claim 2 , wherein to generate the pre-processed data additionally includes to apply a convolution and an activation function to shuffled aligned history data and current frame data.
9. A method comprising:
receiving, at an input block of a first neural network model, a set of data including previous frame data, current frame data, and velocity data, wherein the previous frame data includes one or more previously generated output frames, the current frame data includes output of a raster and lighting stage of a render pipeline, and the velocity data includes motion vectors generated by the render pipeline;
pre-processing the set of data to generate pre-processed data, wherein to pre-process the set of data includes to generate residual motion vectors via a second neural network model;
providing first pre-processed data to a feature extraction network of the first neural network model and second-processed data to an output block of the first neural network model, the first pre-processed data at a first precision and the second pre-processed data at a second precision that is higher than the first precision;
processing the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
outputting tensor data from the feature extraction network to the output block; and
generating an output frame via the output block based on the second pre-processed data from the input block and the tensor data output from the feature extraction network, wherein the output frame is an anti-aliased frame.
10. The method as in claim 9 , wherein generating the pre-processed data includes:
generating the residual motion vectors via the second neural network based on the current frame data, previous frame data, and the motion vectors;
combining the motion vectors and the residual motion vectors to create refined motion vectors;
warping the previous frame data based on the refined motion vectors to generate warped history data;
aligning the warped history data with the current frame data to generate aligned history data; and
shuffling the aligned history data and current frame data from a spatial dimension to a channel dimension.
11. The method as in claim 10 , wherein the channel dimension includes a depth channel or multiple feature map channels and generating pre-processed data additionally includes applying a convolution and an activation function to shuffled aligned history data and current frame data.
12. The method as in claim 10 , wherein generating an output frame via the output block includes generating the output frame via one or more convolution layers of the output block.
13. The method as in claim 10 , wherein generating an output frame via the output block includes to predict a set of per-pixel kernel values and blend weights, filter the current frame data via the per-pixel kernel values, and blend the aligned history data with filtered current frame data.
14. The method as in claim 10 , further comprising upscaling the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution before the current frame data is provided to the input block of the first neural network model.
15. The method as in claim 10 , wherein generating the pre-processed data includes upscaling the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution.
16. A data processing system comprising:
a memory device; and
a graphics processor coupled with the memory device, the graphics processor comprising a set of processing resources configured to perform a supersampling anti-aliasing operation via a mixed precision convolutional neural network, the set of processing resources including circuitry configured to:
receive, at an input block of a first neural network model, a set of data including previous frame data, current frame data, and velocity data, wherein the previous frame data includes one or more previously generated output frames, the current frame data includes output of a raster and lighting stage of a render pipeline, and the velocity data includes motion vectors generated by the render pipeline;
pre-process the set of data to generate pre-processed data, wherein to pre-process the set of data includes to generate residual motion vectors via a second neural network model;
provide first pre-processed data to a feature extraction network of the first neural network model and second-processed data to an output block of the first neural network model, the first pre-processed data at a first precision and the second pre-processed data at a second precision that is higher than the first precision;
process the pre-processed data at the feature extraction network via one or more encoder stages and one or more decoder stages;
output tensor data from the feature extraction network to the output block; and
generate an output frame via the output block based on the second pre-processed data from the input block and the tensor data output from the feature extraction network, wherein the output frame is an anti-aliased frame.
17. The data processing system as in claim 16 , wherein to generate the pre-processed data includes to:
upscale the output of the raster and lighting stage from a first resolution to a second resolution that is higher than the first resolution;
generate the residual motion vectors via the second neural network based on the current frame data, previous frame data, and the motion vectors;
combine the motion vectors and the residual motion vectors to create refined motion vectors;
warp the previous frame data based on the refined motion vectors to generate warped history data;
warp the previous frame data based on the velocity data to generate warped history data;
align the warped history data with the current frame data to generate aligned history data; and
shuffle the aligned history data and current frame data from a spatial dimension to a channel dimension.
18. The data processing system as in claim 17 , wherein the channel dimension includes a depth channel or multiple feature map channels.
19. The data processing system as in claim 17 , wherein to generate the output frame via the output block includes to generate the output frame via one or more convolution layers of the output block.
20. The data processing system as in claim 17 , wherein to generate the output frame via the output block includes to predict a set of per-pixel kernel values and blend weights, filter the current frame data via the per-pixel kernel values, and blend the aligned history data with filtered current frame data.