Machine learning networks for hybrid video compression and corresponding decompression
Innovations in machine learning (“ML”) networks used in video processing scenarios are described. For example, an ML refinement network can be used to refine video after a video decoder has reconstructed the video. Using the ML refinement network for post-processing can mitigate compression artifacts introduced during encoding and otherwise improve the quality of the reconstructed video. Or, as another example, an ML encoder network and ML decoder network can be used, in combination with a core video encoder and core video decoder, for hybrid compression and corresponding decompression. In the hybrid compression, the ML encoder network can transform video before encoding in order to boost rate-distortion performance of the core video encoder. In corresponding decompression, the ML decoder network can enhance reconstructed video after decoding, thereby compensating for transformations applied by the ML encoder network, mitigating compression artifacts, and otherwise improving the quality of the reconstructed video.
1 . A server computer system comprising a processor system and memory, wherein the server computer system is configured to perform operations comprising:
receiving a current unit of input video;
retrieving a given previous unit;
warping the given previous unit to spatially align sample values of the given previous unit with expected locations in a version of the current unit, thereby producing a given warped previous unit;
providing the given warped previous unit to a machine learning (“ML”) encoder network;
with the ML encoder network, transforming the current unit to facilitate preservation of image quality, thereby producing a transformed current unit, wherein, as part of a temporal feedback loop, the transforming the current unit is based at least in part on the given warped previous unit;
encoding the transformed current unit, thereby producing encoded data for the transformed current unit; and
outputting the encoded data as part of a bitstream.
2 . The server computer system of claim 1 , wherein the transforming the current unit also partially compresses the current unit by downsampling the current unit.
3 . The server computer system of claim 1 , wherein the ML encoder network is a convolutional neural network having a U-Net architecture.
4 . The server computer system of claim 1 , wherein the operations further comprise:
decoding the encoded data, thereby producing a decoded current unit; and
with an ML decoder network, enhancing the decoded current unit to compensate for transformations applied by the ML encoder network and mitigate compression artifacts, thereby producing an enhanced current unit.
5 . The server computer system of claim 4 , wherein the enhancing the decoded current unit also partially decompresses the decoded current unit by upsampling the decoded current unit.
6 . The server computer system of claim 4 , wherein the ML decoder network is a convolutional neural network having a U-Net architecture.
7 . The server computer system of claim 4 , wherein the operations further comprise:
storing, in a decoded video buffer, the decoded current unit for use in providing temporal feedback to the ML decoder network.
8 . The server computer system of claim 4 , wherein the given previous unit is a given decoded previous unit retrieved from a decoded video buffer, wherein the given warped previous unit is a given warped, decoded previous unit, and wherein the operations further comprise:
providing the given warped, decoded previous unit to the ML decoder network, wherein the enhancing the decoded current unit is based at least in part on the given warped, decoded previous unit.
9 . The server computer system of claim 4 , wherein the operations further comprise:
storing, in an enhanced video buffer, the enhanced current unit for use in providing temporal feedback to the ML encoder network and the ML decoder network.
10 . The server computer system of claim 4 , wherein the given previous unit is a given enhanced previous unit retrieved from an enhanced video buffer, wherein the given warped previous unit is a given warped, enhanced previous unit, and wherein the operations further comprise:
providing the given warped, enhanced previous unit to the ML decoder network, wherein the enhancing the decoded current unit is based at least in part on the given warped, enhanced previous unit.
11 . A computer system comprising a processor system and memory, wherein the computer system is configured to perform operations comprising:
receiving encoded data for a current unit;
decoding the encoded data, thereby producing a decoded current unit;
retrieving a given previous unit;
warping the given previous unit to spatially align sample values of the given previous unit with expected locations in a version of the current unit, thereby producing a given warped previous unit;
providing the given warped previous unit to a machine learning (“ML”) decoder network; and
with the ML decoder network, enhancing the decoded current unit to compensate for transformations applied by an ML encoder network and mitigate compression artifacts, thereby producing an enhanced current unit, wherein, as part of a temporal feedback loop, the enhancing the decoded current unit is based at least in part on the given warped previous unit.
12 . The computer system of claim 11 , wherein the enhancing the decoded current unit also partially decompresses the decoded current unit by upsampling the decoded current unit.
13 . The computer system of claim 11 , wherein the ML decoder network is a convolutional neural network having a U-Net architecture.
14 . The computer system of claim 11 , wherein the current unit is a frame, a slice, or a tile.
15 . The computer system of claim 11 , wherein the operations further comprise:
storing, in a decoded video buffer, the decoded current unit for use in providing temporal feedback to the ML decoder network.
16 . The computer system of claim 11 , wherein the given previous unit is a given decoded previous unit retrieved from a decoded video buffer, and wherein the given warped previous unit is a given warped, decoded previous unit.
17 . The computer system of claim 11 , wherein the operations further comprise:
storing, in an enhanced video buffer, the enhanced current unit for use in providing temporal feedback to the ML decoder network.
18 . The computer system of claim 11 , wherein the given previous unit is a given enhanced previous unit retrieved from an enhanced video buffer, and wherein the given warped previous unit is a given warped, enhanced previous unit.
19 . The computer system of claim 11 , wherein the operations further comprise:
processing the enhanced current unit for display; and
outputting results of the processing the enhanced current unit for display.
20 . In a computer system, a method of training a machine learning (“ML”) encoder network and an ML decoder network for hybrid compression of video and corresponding decompression, the method comprising:
receiving a current unit of input video;
with an ML encoder network, transforming the current unit to facilitate preservation of image quality, thereby producing a transformed current unit;
encoding the transformed current unit, thereby producing encoded data for the transformed current unit;
decoding the encoded data, thereby producing a decoded current unit;
with an ML decoder network, enhancing the decoded current unit to compensate for transformations applied by the ML encoder network and mitigate compression artifacts, thereby producing an enhanced current unit;
determining feedback based at least in part on differences between the current unit of input video and the enhanced current unit; and
adjusting at least one of the ML encoder network and the ML decoder network based at least in part on the feedback.