Computer-implemented multi-scale machine learning model for the enhancement of compressed video
The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for training and using a multi-scale machine learning model for the enhancement of compressed video. According to some examples, a computer-implemented method includes receiving a video at a content delivery service; performing an encode on a frame of the video by the content delivery service that converts the frame from a pixel domain to a transform domain and back to the pixel domain to generate first pixel values and a first residual for a block of the frame at a first resolution; generating a first set of features, by a machine learning model of the content delivery service, for an input, at a first resolution, of the first pixel values and the first residual of the block; generating a second set of features, by the machine learning model of the content delivery service, for an input, at a second lower resolution, of second pixel values and a second residual of the block; upsampling the second set of features to the first resolution to generate an upsampled second set of features; generating a modified version of the frame based on the first set of features and the upsampled second set of features; and transmitting the modified version of the frame to a frame buffer or from the content delivery service to a viewer device.
1 . A computer-implemented method comprising:
receiving a video at a content delivery service;
performing an encode on a frame of the video by the content delivery service that converts the frame from a pixel domain to a transform domain and back to the pixel domain to generate first pixel values and a first residual for a block of the frame at a first resolution;
generating a first set of features, by a machine learning model of the content delivery service, for an input, at the first resolution, of the first pixel values and the first residual of the block;
generating a second set of features, by the machine learning model of the content delivery service in parallel with the generating the first set of features, for an input, at a second lower resolution, of second pixel values and a second residual of the block;
upsampling the second set of features to the first resolution to generate an upsampled second set of features;
generating a modified version of the frame based on the first set of features and the upsampled second set of features; and
transmitting the modified version of the frame to a frame buffer or from the content delivery service to a viewer device.
2 . The computer-implemented method of claim 1 , further comprising:
generating a third set of features, by the machine learning model of the content delivery service in parallel with the generating the first set of features and the second set of features, for an input, at a third resolution that is lower than the second lower resolution, of third pixel values and a third residual of the block; and
upsampling the third set of features to the first resolution to generate an upsampled third set of features,
wherein the generating the modified version of the frame is based on the first set of features, the upsampled second set of features, and the upsampled third set of features.
3 . The computer-implemented method of claim 1 , wherein the generating the first set of features, generating the second set of features, and generating the modified version of the frame occur within a loop filter of an encoder.
4 . A computer-implemented method comprising:
performing a video coding for a frame of a video that generates first pixel values and a first residual for a block of the frame;
generating a first set of features, by a machine learning model, for an input, at a first resolution, of the first pixel values and the first residual of the block;
generating a second set of features, by the machine learning model in parallel with the generating the first set of features, for an input, at a second lower resolution, of second pixel values and a second residual of the block;
upsampling the second set of features to the first resolution to generate an upsampled second set of features;
generating a modified version of the frame based on the first set of features and the upsampled second set of features; and
transmitting the modified version of the frame to a frame buffer or to a display device.
5 . The computer-implemented method of claim 4 , further comprising:
generating a third set of features, by the machine learning model in parallel with the generating the first set of features and the second set of features, for an input, at a third resolution that is lower than the second lower resolution, of third pixel values and a third residual of the block; and
upsampling the third set of features to the first resolution to generate an upsampled third set of features,
wherein the generating the modified version of the frame is based on the first set of features, the upsampled second set of features, and the upsampled third set of features.
6 . The computer-implemented method of claim 4 , wherein the generating the first set of features, generating the second set of features, and generating the modified version of the frame occur within a loop filter of an encoder.
7 . The computer-implemented method of claim 4 , wherein a viewer device comprises a decoder and the display device, and the generating the first set of features, generating the second set of features, and generating the modified version of the frame occur within a loop filter of the decoder.
8 . The computer-implemented method of claim 7 , further comprising:
determining an indication of a subset of blocks of a frame that are to be processed by a machine learning model of the decoder; and
sending the indication to the decoder to cause the decoder to process the subset of blocks of the frame by the machine learning model of the decoder.
9 . The computer-implemented method of claim 4 , wherein the generating the first set of features, generating the second set of features, and generating the modified version of the frame occur in a post-processor separate from any encoder and any decoder.
10 . The computer-implemented method of claim 4 , further comprising, before the generating the second set of features, downsampling the block from the first resolution to the second lower resolution.
11 . The computer-implemented method of claim 10 , wherein the downsampling comprises performing a strided convolution on the block at the first resolution.
12 . The computer-implemented method of claim 4 , wherein the upsampling comprises interleaving a plurality of channels into one channel.
13 . The computer-implemented method of claim 4 , wherein the generating the modified version of the frame comprises performing a cross-component sample offset operation.
14 . The computer-implemented method of claim 4 , further comprising selecting one of the modified version of the block and another version of the block as input to a cross-component sample offset operation.
15 . A non-transitory computer-readable medium storing code that, when executed by a device, causes the device to perform a method comprising:
performing a video coding for a frame of a video that generates first pixel values and a first residual for a block of the frame;
generating a first set of features, by a machine learning model, for an input, at a first resolution, of the first pixel values and the first residual of the block;
generating a second set of features, by the machine learning model in parallel with the generating the first set of features, for an input, at a second lower resolution, of second pixel values and a second residual of the block;
upsampling the second set of features to the first resolution to generate an upsampled second set of features;
generating a modified version of the frame based on the first set of features and the upsampled second set of features; and
transmitting the modified version of the frame to a frame buffer or to a display device.
16 . The non-transitory computer-readable medium of claim 15 ,
wherein the method further comprises:
generating a third set of features, by the machine learning model in parallel with the generating the first set of features and the second set of features, for an input, at a third resolution that is lower than the second lower resolution, of third pixel values and a third residual of the block; and
upsampling the third set of features to the first resolution to generate an upsampled third set of features,
wherein the generating the modified version of the frame is based on the first set of features, the upsampled second set of features, and the upsampled third set of features.
17 . The non-transitory computer-readable medium of claim 15 , wherein the generating the first set of features, generating the second set of features, and generating the modified version of the frame occur within a loop filter of an encoder.
18 . The non-transitory computer-readable medium of claim 15 , wherein the device comprises a decoder and the display device, and the generating the first set of features, generating the second set of features, and generating the modified version of the frame occur within a loop filter of the decoder.
19 . The non-transitory computer-readable medium of claim 15 , wherein the generating the first set of features, generating the second set of features, and generating the modified version of the frame occur in a post-processor separate from any encoder of the device and any decoder of the device.
20 . The non-transitory computer-readable medium of claim 15 , wherein the generating the modified version of the frame comprises performing a cross-component sample offset operation.