Augmenting temporal anti-aliasing with a neural network for history validation
An apparatus to facilitate augmenting temporal anti-aliasing with a neural network for history validation is disclosed. The apparatus includes a set of processing resources configured to perform augmented temporal anti-aliasing (TAA), the set of processing resources including circuitry configured to: receive, at a history validation neural network, inputs for a current pixel of a current frame and a reprojected pixel corresponding to the current pixel, the reprojected pixel originating from history data of the current frame; generate, using an output of the history validation neural network, a validated color for the current pixel based on current color data corresponding to the current pixel and history color data corresponding to the reprojected pixel; render an output frame using the validated color; and add the output frame to the history data.
1. An apparatus comprising:
a set of processing resources configured to perform augmented temporal anti-aliasing (TAA), the set of processing resources including circuitry configured to:
receive, at a history validation neural network, inputs for a current pixel of a current frame and a reprojected pixel corresponding to the current pixel, the reprojected pixel originating from history data of the current frame, wherein the inputs comprise data corresponding to current color data of the current pixel, current frame depth of the current pixel, current frame auxiliary buffers of the current pixel, history color data of the reprojected pixel, and a time delta between frames comprising the current pixel and the reprojected pixel;
generate, using an output of the history validation neural network, a validated color for the current pixel based on the current color data corresponding to the current pixel and the history color data corresponding to the reprojected pixel;
render an output frame using the validated color; and
add the output frame to the history data.
2. The apparatus of claim 1 , wherein the output of the history validation neural network comprises the validated color for the current pixel.
3. The apparatus of claim 1 , wherein the output of the history validation neural network comprises a validated historical frame color and a temporal blend coefficient for the current pixel, wherein the temporal blend coefficient indicates how to blend the validated color with the current color data of the current pixel.
4. The apparatus of claim 1 , wherein the history validation neural network comprises a small multilayered perceptron.
5. The apparatus of claim 1 , wherein the inputs comprise a K×K neighborhood of the current color data of the current pixel, the K×K neighborhood of the current frame depth of the current pixel, the K×K neighborhood of the current frame auxiliary buffers of the current pixel, a R×R neighborhood of the history color data of the reprojected pixel, and the time delta between the frames comprising the current pixel and the reprojected pixel.
6. The apparatus of claim 5 , wherein the reprojected pixel originates from more than one previous frame of the current frame, and wherein the R×R neighborhood in the current frame of accumulated history color data of the reprojected pixel comprises the accumulated history color data of the reprojected pixel from the more than one previous frame.
7. The apparatus of claim 5 , wherein the time delta between the frames provides an association to a real world clock for utilization in a rate of averaging implemented by the history validation neural network.
8. The apparatus of claim 1 , wherein the history validation neural network is trained per a set of content or a portion of the set of content.
9. A method comprising:
receiving, by a processing resource hosting a history validation neural network, inputs for the history validation neural network, wherein the inputs are for a current pixel of a current frame and a reprojected pixel corresponding to the current pixel, and wherein the reprojected pixel originating from history data of the current frame, wherein the inputs comprise data corresponding to current color data of the current pixel, current frame depth of the current pixel, current frame auxiliary buffers of the current pixel, history color data of the reprojected pixel, and a time delta between frames comprising the current pixel and the reprojected pixel;
generating, by the processing resource using an output of the history validation neural network, a validated color for the current pixel based on the current color data corresponding to the current pixel and he history color data corresponding to the reprojected pixel;
rendering, by the processing resource, an output frame using the validated color; and
adding, by the processing resource, the output frame to the history data.
10. The method of claim 9 , wherein the output of the history validation neural network comprises the validated color for the current pixel.
11. The method of claim 9 , wherein the output of the history validation neural network comprises a validated historical frame color and a temporal blend coefficient for the current pixel, wherein the temporal blend coefficient indicates how to blend the validated color with the current color data of the current pixel.
12. The method of claim 9 , wherein the history validation neural network comprises a small multilayered perceptron.
13. The method of claim 9 , wherein the inputs comprise a K×K neighborhood of the current color data of the current pixel, the K×K neighborhood of the current frame depth of the current pixel, the K×K neighborhood of the current frame auxiliary buffers of the current pixel, a R×R neighborhood of the history color data of the reprojected pixel, and the time delta between the frames comprising the current pixel and the reprojected pixel.
14. The method of claim 13 , wherein the reprojected pixel originates from more than one previous frame of the current frame, and wherein the R×R neighborhood in the current frame of accumulated history color data of the reprojected pixel comprises the accumulated history color data of the reprojected pixel from the more than one previous frame.
15. The method of claim 13 , wherein the time delta between the frames provides an association to a real world clock for utilization in a rate of averaging implemented by the history validation neural network.
16. A system comprising:
a memory device; and
a graphics processor coupled with the memory device, the graphics processor comprising a set of processing resources to perform augmented temporal anti-aliasing (TAA), the set of processing resources including circuitry configured to:
receive, at a history validation neural network, inputs for a current pixel of a current frame and a reprojected pixel corresponding to the current pixel, the reprojected pixel originating from history data of the current frame, wherein the inputs comprise data corresponding to current color data of the current pixel, current frame depth of the current pixel, current frame auxiliary buffers of the current pixel, history color data of the reprojected pixel, and a time delta between frames comprising the current pixel and the reprojected pixel;
generate, using an output of the history validation neural network, a validated color for the current pixel based on the current color data corresponding to the current pixel and the history color data corresponding to the reprojected pixel;
render an output frame using the validated color; and
add the output frame to the history data.
17. The system of claim 16 , wherein the output of the history validation neural network comprises the validated color for the current pixel.
18. The system of claim 16 , wherein the output of the history validation neural network comprises a validated historical frame color and a temporal blend coefficient for the current pixel, wherein the temporal blend coefficient indicates how to blend the validated color with the current color data of the current pixel.
19. The system of claim 16 , wherein the history validation neural network comprises a small multilayered perceptron.
20. The system of claim 16 , wherein the inputs comprise a K×K neighborhood of the current color data of the current pixel, the K×K neighborhood of the current frame depth of the current pixel, the K×K neighborhood of the current frame auxiliary buffers of the current pixel, a R×R neighborhood of the history color data of the reprojected pixel, and the time delta between the frames comprising the current pixel and the reprojected pixel.
21. A non-transitory computer-readable storage medium having stored thereon executable computer program instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving, by the one or more processors hosting a history validation neural network, inputs for the history validation neural network, wherein the inputs are for a current pixel of a current frame and a reprojected pixel corresponding to the current pixel, and wherein the reprojected pixel originating from history data of the current frame, wherein the inputs comprise data corresponding to current color data of the current pixel, current frame depth of the current pixel, current frame auxiliary buffers of the current pixel, history color data of the reprojected pixel, and a time delta between frames comprising the current pixel and the reprojected pixel;
generating, by the one or more processors using an output of the history validation neural network, a validated color for the current pixel based on the current color data corresponding to the current pixel and the history color data corresponding to the reprojected pixel;
rendering, by the one or more processors, an output frame using the validated color; and
adding, by the one or more processors, the output frame to the history data.
22. The non-transitory computer-readable storage medium of claim 21 , wherein the output of the history validation neural network comprises the validated color for the current pixel.
23. The non-transitory computer-readable storage medium of claim 21 , wherein the output of the history validation neural network comprises a validated historical frame color and a temporal blend coefficient for the current pixel, wherein the temporal blend coefficient indicates how to blend the validated color with the current color data of the current pixel.
24. The non-transitory computer-readable storage medium of claim 21 , wherein the history validation neural network comprises a small multilayered perceptron.
25. The non-transitory computer-readable storage medium of claim 21 , wherein the inputs comprise a K×K neighborhood of the current color data of the current pixel, the K×K neighborhood of the current frame depth of the current pixel, the K×K neighborhood of the current frame auxiliary buffers of the current pixel, a R×R neighborhood of the history color data of the reprojected pixel, and the time delta between the frames comprising the current pixel and the reprojected pixel.