3D transform and inter prediction for video coding
Methods and devices for video coding using inter prediction and three-dimensional transform. A sequence of pictures in the video is partitioned into coding cuboids. A predicted cuboid is generated for one of the coding cuboids using interprediction, including at least one motion vector and at least one reference picture index. The difference between the coding cuboid and the predicted cuboid results in a cuboid of residuals, which is then spectrally transformed using a three-dimensional transform to create a cuboid of transform domain coefficients, which are then entropy encoded. The decoding process generates the same predicted cuboid and reconstructs the cuboid of pixels using the predicted cuboid and a cuboid of reconstructed residuals.
1. A method of decoding a video from a bitstream, the video including a sequence of pictures in a coding order, the method comprising:
decoding a partition mode from the bitstream, the partition mode defining the partitioning of a portion of the sequence of pictures into a plurality of rectangular coding cuboids, at least one rectangular coding cuboid including portions of two or more consecutive pictures in the coding order;
decoding, from the bitstream, inter-prediction information associated with the at least one rectangular coding cuboid, including at least one motion vector and at least one reference picture index;
generating a predicted cuboid for said at least one rectangular coding cuboid based on the at least one motion vector and the at least one reference picture index;
reconstructing a cuboid of transform domain coefficients from the bitstream;
dequantizing and three-dimensional inverse transforming the cuboid of transform domain coefficients to produce a reconstructed residual cuboid; and
reconstructing the pixels of said at least one rectangular coding cuboid as the sum of the predicted cuboid and the reconstructed residual cuboid,
whereby the three-dimensional inverse transforming improves data compression through exploiting temporal redundancy remaining in the reconstructed residual cuboid despite the use of inter-prediction.
2. The method claimed in claim 1 , wherein the at least one motion vector comprises a single motion vector, the at least one reference picture index comprises a single reference picture index, and wherein generating the predicted cuboid includes copying a reference cuboid of pixel values specified by the single motion vector and the single reference picture index, wherein the reference cuboid includes portions of at least two previously-reconstructed consecutive pictures in the coding order.
3. The method claimed in claim 1 , wherein the at least one motion vector comprises a single motion vector, the at least one reference picture index comprises a single reference picture index, and wherein generating the predicted cuboid comprises, for each portion of the two or more consecutive pictures,
copying a block of pixel values in a single reference picture specified by the single reference picture index, wherein block of pixels for each portion of the two or more consecutive pictures is specified by a weighted motion vector, wherein the weighted motion vector comprises the single motion vector multiplied by a weighting factor.
4. The method claimed in claim 3 , wherein the weighting factor is based on the difference in position in the coding order between the portion of the two or more consecutive pictures being predicted and the single reference picture.
5. The method claimed in claim 4 , wherein the weighting factor is further based on the distance between a first of the two or more consecutive pictures and the single reference picture.
6. The method claimed in claim 1 , wherein the at least one motion vector includes a respective motion vector associated with each portion, and wherein generating the predicted cuboid comprises, for each portion of the two or more consecutive pictures, copying a block of pixel values in a respective reconstructed picture specified by the respective motion vector associated with that portion.
7. The method claimed in claim 1 , wherein three-dimensional inverse transforming the cuboid of transform domain coefficients comprises applying a one-dimensional inverse transform to a series of coefficients having the same row and column index in consecutive blocks of the cuboid of transform domain coefficients to produce a partially inverse transformed cuboid, and then applying a two-dimensional inverse transform to the partially inverse transformed cuboid to produce the reconstructed residual cuboid.
8. A decoder for decoding a video from a bitstream, the video including a sequence of pictures in a coding order, the decoder comprising:
a processor;
memory; and
a decoding application containing instructions executable by the processor that, when executed, cause the processor to
decode a partition mode from the bitstream, the partition mode defining the partitioning of a portion of the sequence of pictures into a plurality of rectangular coding cuboids, at least one rectangular coding cuboid including two or more consecutive pictures in the coding order;
decode, from the bitstream, inter-prediction information associated with the at least one rectangular coding cuboid, including at least one motion vector and at least one reference picture index;
generate a predicted cuboid for said at least one rectangular coding cuboid based on the at least one motion vector and the at least one reference picture index;
reconstruct a cuboid of transform domain coefficients from the bitstream;
dequantize and three-dimensional inverse transforming the cuboid of transform domain coefficients to produce a reconstructed residual cuboid; and
reconstruct the pixels of said at least one rectangular coding cuboid as the sum of the predicted cuboid and the reconstructed residual cuboid,
whereby the three-dimensional inverse transforming improves data compression through exploiting temporal redundancy remaining in the reconstructed residual cuboid despite the use of inter-prediction.
9. The decoder claimed in claim 8 , wherein the at least one motion vector comprises a single motion vector, the at least one reference picture index comprises a single reference picture index, and wherein the instructions, when executed, cause the processor to generate the predicted cuboid by copying a reference cuboid of pixel values specified by the single motion vector and the single reference picture index, wherein the reference cuboid includes portions of at least two previously-reconstructed consecutive pictures in the coding order.
10. The decoder claimed in claim 8 , wherein the at least one motion vector comprises a single motion vector, the at least one reference picture index comprises a single reference picture index, and wherein the instructions, when executed, cause the processor to generate the predicted cuboid by, for each portion of the two or more consecutive pictures,
copying a block of pixel values in a single reference picture specified by the single reference picture index, wherein block of pixels for each portion of the two or more consecutive pictures is specified by a weighted motion vector, wherein the weighted motion vector comprises the single motion vector multiplied by a weighting factor.
11. The decoder claimed in claim 10 , wherein the weighting factor is based on the difference in position in the coding order between the portion of the two or more consecutive pictures being predicted and the single reference picture.
12. The decoder claimed in claim 11 , wherein the weighting factor is further based on the distance between a first of the two or more consecutive pictures and the single reference picture.
13. The decoder claimed in claim 8 , wherein the at least one motion vector includes a respective motion vector associated with each portion, and wherein the instructions, when executed, cause the processor to generate the predicted cuboid by, for each portion of the two or more consecutive pictures, copying a block of pixel values in a respective reconstructed picture specified by the respective motion vector associated with that portion.
14. The decoder claimed in claim 8 , and wherein the instructions, when executed, cause the processor to three-dimensional inverse transform the cuboid of transform domain coefficients by applying a one-dimensional inverse transform to a series of coefficients having the same row and column index in consecutive blocks of the cuboid of transform domain coefficients to produce a partially inverse transformed cuboid, and then applying a two-dimensional inverse transform to the partially inverse transformed cuboid to produce the reconstructed residual cuboid.
15. A non-transitory processor-readable medium storing processor-executable instructions for decoding a video from a bitstream, the video including a sequence of pictures in a coding order, wherein the processor-executable instructions, when executed by a processor in the video or image decoder, cause the processor to:
decode a partition mode from the bitstream, the partition mode defining the partitioning of a portion of the sequence of pictures into a plurality of rectangular coding cuboids, at least one rectangular coding cuboid including two or more consecutive pictures in the coding order;
decode, from the bitstream, inter-prediction information associated with the at least one rectangular coding cuboid, including at least one motion vector and at least one reference picture index;
generate a predicted cuboid for said at least one rectangular coding cuboid based on the at least one motion vector and the at least one reference picture index;
reconstruct a cuboid of transform domain coefficients from the bitstream;
dequantize and three-dimensional inverse transforming the cuboid of transform domain coefficients to produce a reconstructed residual cuboid; and
reconstruct the pixels of said at least one rectangular coding cuboid as the sum of the predicted cuboid and the reconstructed residual cuboid,
whereby the three-dimensional inverse transforming improves data compression through exploiting temporal redundancy remaining in the reconstructed residual cuboid despite the use of inter-prediction.
16. A method of encoding a video in a video encoder to produce a bitstream, the video including a sequence of pictures in a coding order, the sequence of pictures being partitioned into a plurality of rectangular coding cuboids, at least one rectangular coding cuboid including portions of two or more consecutive pictures in the coding order, the method comprising:
selecting inter-prediction information for the at least one rectangular coding cuboid, including at least one motion vector and at least one reference picture index;
generating a predicted cuboid for said at least one rectangular coding cuboid based on the at least one motion vector and the at least one reference picture index;
determining a cuboid of residuals by subtracting the predicted cuboid from said at least one rectangular coding cuboid;
three-dimensional spectral transforming and quantizing the cuboid of residuals to generate a cuboid of transform domain coefficients; and
entropy encoding the cuboid of transform domain coefficients and the inter-prediction information,
whereby the three-dimensional spectral transforming improves data compression through exploiting temporal redundancy remaining in the cuboid of residuals despite the use of inter-prediction.
17. The method claimed in claim 16 , wherein the at least one motion vector comprises a single motion vector, the at least one reference picture index comprises a single reference picture index, and wherein generating the predicted cuboid includes copying a reference cuboid of pixel values specified by the single motion vector and the single reference picture index, wherein the reference cuboid includes portions of at least two previously-reconstructed consecutive pictures in the coding order.
18. The method claimed in claim 16 , wherein the at least one motion vector comprises a single motion vector, the at least one reference picture index comprises a single reference picture index, and wherein generating the predicted cuboid comprises, for each portion of the two or more consecutive pictures,
copying a block of pixel values in a single reference picture specified by the single reference picture index, wherein block of pixels for each portion of the two or more consecutive pictures is specified by a weighted motion vector, wherein the weighted motion vector comprises the single motion vector multiplied by a weighting factor.
19. The method claimed in claim 18 , wherein the weighting factor is based on the difference in position in the coding order between the portion of the two or more consecutive pictures being predicted and the single reference picture.
20. The method claimed in claim 19 , wherein the weighting factor is further based on the distance between a first of the two or more consecutive pictures and the single reference picture.
21. The method claimed in claim 16 , wherein the at least one motion vector includes a respective motion vector associated with each portion, and wherein generating the predicted cuboid comprises, for each portion of the two or more consecutive pictures, copying a block of pixel values in a respective reconstructed picture specified by the respective motion vector associated with that portion.
22. The method claimed in claim 16 , wherein the cuboid of residuals includes a series of blocks of residuals corresponding to the portions of two or more consecutive pictures, and wherein three-dimensional spectral transforming the cuboid of residuals comprises applying a two-dimensional inverse transform to each of the blocks in the series to produce a two-dimensional transformed cuboid of coefficients, and then applying a one-dimensional transform to a sequence of coefficients having the same row and column index in the series of blocks to produce the cuboid of transform domain coefficients.
23. An encoder for encoding a video to produce a bitstream of encoded data, the video including a sequence of pictures in a coding order, the sequence of pictures being partitioned into a plurality of rectangular coding cuboids, at least one rectangular coding cuboid including portions of two or more consecutive pictures in the coding order, the encoder comprising:
a processor;
memory; and
an encoding application containing instructions executable by the processor that, when executed, cause the processor to
select inter-prediction information for the at least one rectangular coding cuboid, including at least one motion vector and at least one reference picture index;
generate a predicted cuboid for said at least one rectangular coding cuboid based on the at least one motion vector and the at least one reference picture index;
determine a cuboid of residuals by subtracting the predicted cuboid from said at least one rectangular coding cuboid;
three-dimensional spectral transform and quantize the cuboid of residuals to generate a cuboid of transform domain coefficients; and
entropy encode the cuboid of transform domain coefficients and the inter-prediction information,
whereby the three-dimensional spectral transforming improves data compression through exploiting temporal redundancy remaining in the cuboid of residuals despite the use of inter-prediction.
24. A non-transitory processor-readable medium storing processor-executable instructions for encoding an image in a video or image encoder to produce a bitstream, the video including a sequence of pictures in a coding order, the sequence of pictures being partitioned into a plurality of rectangular coding cuboids, at least one rectangular coding cuboid including portions of two or more consecutive pictures in the coding order, wherein the processor-executable instructions, when executed by a processor in the video or image encoder, cause the processor to:
select inter-prediction information for the at least one rectangular coding cuboid, including at least one motion vector and at least one reference picture index;
generate a predicted cuboid for said at least one rectangular coding cuboid based on the at least one motion vector and the at least one reference picture index;
determine a cuboid of residuals by subtracting the predicted cuboid from said at least one rectangular coding cuboid;
three-dimensional spectral transform and quantize the cuboid of residuals to generate a cuboid of transform domain coefficients; and
entropy encode the cuboid of transform domain coefficients and the inter-prediction information,
whereby the three-dimensional spectral transforming improves data compression through exploiting temporal redundancy remaining in the cuboid of residuals despite the use of inter-prediction.