Video upsampling using one or more neural networks
Apparatuses, systems, and techniques to enhance video are disclosed. In at least one embodiment, one or more neural networks are used to create a higher resolution video using upsampled frames from a lower resolution video.
1. A system-on-chip (SOC), comprising:
a graphics core comprising:
an instruction cache;
a cache/shared memory;
floating point logic units to perform 16-bit, 32-bit, or 64-bit floating point operations;
integer logic units; and
matrix processing units (MPUs) to perform half-precision floating point or 8-bit integer operations;
memory; and
a system to upsample a lower-resolution frame, blend the upsampled frame with data from a previous frame, upsample the lower-resolution frame using a neural network to infer an upsampled output frame, and blend the upsampled output frame with data from a previous frame.
2. The SOC of claim 1 , wherein the system is to upsample from 1080p to 4k resolution.
3. The SOC of claim 1 , wherein the system is to upsample by taking into account jitter.
4. The SOC of claim 1 , wherein the system is to further warp the previous frame.
5. The SOC of claim 1 , wherein the previous frame is in a color space that includes a luma value and two chroma values.
6. The SOC of claim 1 , wherein the system is to blend the upsampled frame with data from a previous frame using a luma channel.
7. The SOC of claim 1 , wherein the system is to blend the upsampled output frame with data from a previous frame using a luma channel.
8. The SOC of claim 1 , wherein the lower-resolution frame is a video frame.
9. The SOC of claim 1 , wherein the system is to perform anti-aliasing as part of an upsampling process.
10. The SOC of claim 1 , wherein the system is to perform smoothing as part of an upsampling process.
11. The SOC of claim 1 , wherein the previous frame was upsampled using the neural network.
12. A processor, comprising:
a graphics core comprising:
an instruction cache;
a cache/shared memory;
floating point logic units to perform 16-bit, 32-bit, or 64-bit floating point operations;
integer logic units; and
matrix processing units (MPUs) to perform half-precision floating point and 8-bit integer operations;
memory; and
wherein the processor is to upsample a lower-resolution frame, blend the upsampled frame with data from a previous frame, upsample the lower-resolution frame using a neural network to infer an upsampled output frame, and blend the upsampled output frame with data from a previous frame.
13. The processor of claim 12 , wherein the processor is to upsample from 1080p to 4k resolution.
14. The processor of claim 12 , wherein the processor is to upsample by taking into account jitter.
15. The processor of claim 12 , wherein the processor is to further warp the previous frame.
16. The processor of claim 12 , wherein the previous frame is in a color space that includes a luma value and two chroma values.
17. The processor of claim 12 , wherein the processor is to blend the upsampled frame with data from a previous frame using a luma channel.
18. The processor of claim 12 , wherein the processor is to blend the upsampled output frame with data from a previous frame using a luma channel.
19. The processor of claim 12 , wherein the lower-resolution frame is a video frame.
20. The processor of claim 12 , wherein the processor is to perform anti-aliasing as part of an upsampling process.
21. The processor of claim 12 , wherein the processor is to perform smoothing as part of an upsampling process.
22. The processor of claim 12 , wherein the previous frame was upsampled using the neural network.
23. A method, comprising:
using a system-on-chip (SOC) to upsample a lower-resolution frame, wherein the SOC comprises:
a graphics core comprising:
an instruction cache;
a cache/shared memory;
floating point logic units to perform 16-bit, 32-bit, or 64-bit floating point operations;
integer logic units; and
matrix processing units (MPUs) to perform half-precision floating point and 8-bit integer operations; and
memory; and
blending, by the SOC, the upsampled frame with data from a previous frame;
upsampling, by the SOC, another lower-resolution frame using a neural network to infer an upsampled output frame; and
blending, by the SOC, the upsampled output frame with data from another previous frame.
24. The method of claim 23 , wherein upsampling the lower-resolution frame and the other lower-resolution frame is from 1080p to 4k resolution.
25. The method of claim 23 , wherein upsampling the lower-resolution frame or the other lower-resolution frame takes into account jitter.
26. The method of claim 23 , further comprising warping the previous frame or other previous frame.
27. The method of claim 23 , wherein the previous frame or the other previous frame is in a color space that includes a luma value and two chroma values.
28. The method of claim 23 , wherein blending the upsampled frame with the data from the previous frame uses a luma channel.
29. The method of claim 23 , wherein blending the upsampled frame with the data from the previous frame or blending the other upsampled frame with the data from the other previous frame uses a luma channel.
30. The method of claim 23 , wherein the lower-resolution frame or the other lower-resolution frame is a video frame.
31. The method of claim 23 , wherein upsampling comprises anti-aliasing.
32. The method of claim 23 , wherein upsampling comprises smoothing.
33. The method of claim 23 , wherein the previous frame was upsampled using the neural network.