Video codec assisted real-time video enhancement using deep learning
Techniques related to accelerated video enhancement using deep learning selectively applied based on video codec information are discussed. Such techniques include applying a deep learning video enhancement network selectively to decoded non-skip blocks that are in low quantization parameter frames, bypassing the deep learning network for decoded skip blocks in low quantization parameter frames, and applying non-deep learning video enhancement to high quantization parameter frames.
1 . A method, comprising:
determining whether a quantization parameter of a first image is smaller than a threshold, the first image having a first resolution and comprising a first pixel block and a second pixel block;
in response to determining that the quantization parameter of the first image is smaller than the threshold:
generating, by a deep learning network, a third pixel block from the first pixel block, and
retrieving a fourth pixel block from a second image based on the second pixel block, wherein the fourth pixel block is a portion of the second image, and the second image is generated before the first image is generate; and
generating a third image by merging the third pixel block and the fourth pixel block, wherein the third image has a second resolution that is higher than the first resolution.
2 . The method of claim 1 , further comprising:
in response to determining that the quantization parameter of the first image is not smaller than the threshold, generating a fifth pixel block by applying interpolation to the first pixel block or the second pixel block, the fifth pixel block having the second resolution.
3 . The method of claim 1 , wherein retrieving the fourth pixel block from the second image based on the second pixel block comprises:
determining a size or position of the fourth pixel block based on a size or position of the second pixel block.
4 . The method of claim 1 , wherein the third image and the first image have a same time instance in a video.
5 . The method of claim 1 , wherein the second image is temporally adjacent to the first image in a video.
6 . The method of claim 5 , further comprising:
identifying the second image from the video based on a motion vector of the second pixel block.
7 . The method of claim 1 , wherein the deep learning network comprises a deep learning super-resolution network.
8 . One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising:
determining whether a quantization parameter of a first image is smaller than a threshold, the first image having a first resolution and comprising a first pixel block and a second pixel block;
in response to determining that the quantization parameter of the first image is smaller than the threshold:
generating, by a deep learning network, a third pixel block from the first pixel block, and
retrieving a fourth pixel block from a second image based on the second pixel block, wherein the fourth pixel block is a portion of the second image, and the second image is generated before the first image is generate; and
generating a third image by merging the third pixel block and the fourth pixel block, wherein the third image has a second resolution that is higher than the first resolution.
9 . The one or more non-transitory computer-readable media of claim 8 , wherein the operations further comprise:
in response to determining that the quantization parameter of the first image is not smaller than the threshold, generating a fifth pixel block by applying interpolation to the first pixel block or the second pixel block, the fifth pixel block having the second resolution.
10 . The one or more non-transitory computer-readable media of claim 8 , wherein retrieving the fourth pixel block from the second image based on the second pixel block comprises:
determining a size or position of the fourth pixel block based on a size or position of the second pixel block.
11 . The one or more non-transitory computer-readable media of claim 8 , wherein the third image and the first image have a same time instance in a video.
12 . The one or more non-transitory computer-readable media of claim 8 , wherein the second image is temporally adjacent to the first image in a video.
13 . The one or more non-transitory computer-readable media of claim 12 , wherein the operations further comprise:
identifying the second image from the video based on a motion vector of the second pixel block.
14 . The one or more non-transitory computer-readable media of claim 8 , wherein the deep learning network comprises a deep learning super-resolution network.
15 . An apparatus, comprising:
a computer processor for executing computer program instructions; and
a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising:
determining whether a quantization parameter of a first image is smaller than a threshold, the first image having a first resolution and comprising a first pixel block and a second pixel block,
in response to determining that the quantization parameter of the first image is smaller than the threshold:
generating, by a deep learning network, a third pixel block from the first pixel block, and
retrieving a fourth pixel block from a second image based on the second pixel block, wherein the fourth pixel block is a portion of the second image, and the second image is generated before the first image is generate, and
generating a third image by merging the third pixel block and the fourth pixel block, wherein the third image has a second resolution that is higher than the first resolution.
16 . The apparatus of claim 15 , wherein the operations further comprise:
in response to determining that the quantization parameter of the first image is not smaller than the threshold, generating a fifth pixel block by applying interpolation to the first pixel block or the second pixel block, the fifth pixel block having the second resolution.
17 . The apparatus of claim 15 , wherein retrieving the fourth pixel block from the second image based on the second pixel block comprises:
determining a size or position of the fourth pixel block based on a size or position of the second pixel block.
18 . The apparatus of claim 15 , wherein the third image and the first image have a same time instance in a video.
19 . The apparatus of claim 15 , wherein the second image is temporally adjacent to the first image in a video.
20 . The apparatus of claim 19 , wherein the operations further comprise:
identifying the second image from the video based on a motion vector of the second pixel block.