Computer-implemented method and apparatus for video coding using super-resolution restoration with residual frame coding
The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for video coding using super-resolution restoration with residual frame coding. According to some examples, a computer-implemented method includes receiving a coded frame of a video; performing a video coding on the coded frame of the video to generate a resultant for the coded frame at a second lower resolution than a first resolution; upsampling the resultant in at least a vertical direction to a higher resolution than the second lower resolution to generate an upsampled resultant; generating a decoded frame based on at least the upsampled resultant; and transmitting the decoded frame to a frame buffer or to a display device.
1. A computer-implemented method comprising:
receiving a video at a content delivery service;
downsampling a frame of the video from a first resolution to a second lower resolution in a vertical direction and a horizontal direction;
performing a video coding on the frame of the video by the content delivery service that coverts the frame from a pixel domain to a transform domain and back to the pixel domain to generate a resultant for the frame at the second lower resolution;
upsampling the resultant in the vertical direction and the horizontal direction to a higher resolution than the second lower resolution to generate an upsampled resultant;
performing an entropy encode of the frame based on at least the upsampled resultant to generate a coded frame;
transmitting the coded frame from the content delivery service to a decoder;
generating a reference frame at the higher resolution for the frame based on the upsampled resultant;
storing the reference frame at the higher resolution in a frame buffer; and
storing a corresponding decoded motion vector at the second lower resolution for the frame in the frame buffer.
2. The computer-implemented method of claim 1 , wherein
the storing the reference frame at the higher resolution and
the storing the corresponding decoded motion vector at the second lower resolution for the frame are in a single frame unit in the frame buffer.
3. The computer-implemented method of claim 1 , wherein the upsampling comprises generating the upsampled resultant by a machine learning model.
4. A computer-implemented method comprising:
receiving a coded frame of a video;
performing a video coding on the coded frame of the video to generate a resultant for the coded frame at a second lower resolution than a first resolution;
upsampling the resultant in at least a vertical direction to a higher resolution than the second lower resolution to generate an upsampled resultant;
generating a decoded frame at the higher resolution based on at least the upsampled resultant;
causing a store of the decoded frame at the higher resolution in a frame buffer; and
causing a store of a corresponding decoded motion vector at the second lower resolution for the coded frame in the frame buffer.
5. The computer-implemented method of claim 4 , wherein the causing the store of the decoded frame at the higher resolution and the causing the store of the corresponding decoded motion vector at the second lower resolution for the coded frame are in a single frame unit in the frame buffer.
6. The computer-implemented method of claim 4 , wherein the frame buffer does not include a corresponding decoded motion vector at the higher resolution for the coded frame.
7. The computer-implemented method of claim 4 , wherein the upsampling comprises generating the upsampled resultant by a machine learning model.
8. The computer-implemented method of claim 7 , wherein the generating the decoded frame comprises performing a residual operation based on the upsampled resultant at the higher resolution to generate the decoded frame.
9. The computer-implemented method of claim 4 , wherein the generating the decoded frame comprises performing a residual operation based on the upsampled resultant at the higher resolution to generate the decoded frame.
10. The computer-implemented method of claim 4 , wherein a viewer device comprises a decoder including the frame buffer, and the upsampling the resultant occurs within a postprocessing operation of the decoder.
11. The computer-implemented method of claim 10 , wherein the postprocessing operation comprises the upsampling in at least the vertical direction and a loop restoration operation.
12. The computer-implemented method of claim 11 , wherein the postprocessing operation further comprises performing a residual operation based on the upsampled resultant at the higher resolution to generate the decoded frame.
13. The computer-implemented method of claim 4 , wherein the upsampling the resultant in at least the vertical direction to the higher resolution occurs within a decoder in a super-resolution mode.
14. The computer-implemented method of claim 4 , wherein a header for the video comprises a super-resolution scaling parameter for the vertical direction.
15. An apparatus comprising:
a coupling to a display; and
a video decoder to:
receive a coded frame of a video,
perform a video coding on the coded frame of the video to generate a resultant for the coded frame at a second lower resolution than a first resolution,
upsample the resultant in at least a vertical direction to a higher resolution than the second lower resolution to generate an upsampled resultant,
generate a decoded frame at the higher resolution based on the upsampled resultant,
store the decoded frame at the higher resolution in a frame buffer, and
store a corresponding decoded motion vector at the second lower resolution for the coded frame in the frame buffer.
16. The apparatus of claim 15 , wherein the store of the decoded frame at the higher resolution and the store of the corresponding decoded motion vector at the second lower resolution for the coded frame are in a single frame unit in the frame buffer.
17. The apparatus of claim 16 , wherein the frame buffer does not include a corresponding decoded motion vector at the higher resolution for the coded frame.
18. The apparatus of claim 15 , wherein the video decoder is to generate the upsampled resultant by a machine learning model.
19. The apparatus of claim 15 , wherein the video decoder is to upsample the resultant in at least the vertical direction to the higher resolution when the video decoder is in a super-resolution mode.
20. The apparatus of claim 15 , wherein the video decoder is further to read a header for the video that comprises a super-resolution scaling parameter for the vertical direction.