Joint spatio-temporal prediction for video coding
Systems and methods are described for joint spatio-temporal prediction (JSTP) during video coding. A given block of pixels is simultaneously predicted via inter frame prediction and intra frame prediction. The joint spatio-temporal prediction provides a new JSTP coding mode besides inter and intra modes in coding schemata such as the H.264/AVC video coding standard. In one implementation, subband decomposition obtains a high-pass component of inter frame prediction of a block and a low-pass component of intra frame prediction of the same block. The high-pass component of the inter prediction and the low-pass component of the intra prediction are combined to produce the joint spatio-temporal prediction. In one implementation, a difference between the original block and the JSTP prediction is coded in a bitstream to represent the block.
1. A method implemented by a computing device, comprising:
receiving, at the computing device, a bitstream representing a video sequence, the bitstream including a motion vector, a direction vector, and a difference that is to be added to a joint spatio-temporally predicted block to obtain a block of a frame in the video sequence;
determining, by the computing device and at least partly in response to receiving the bitstream, the joint spatio-temporally predicted block by:
determining an inter frame prediction of the block based at least in part on the motion vector;
decomposing, applying a subband decomposition, the inter frame prediction into a high-pass component and a low-pass component; and
determining an intra frame prediction of the block based at least in part on the direction vector;
decomposing, applying a subband decomposition, the intra frame prediction into a high-pass component and a low-pass component; and
combining the low-pass component of the intra frame prediction with the high-pass component of the inter frame prediction to obtain the joint spatio-temporally predicted block; and
adding, by the computing device, the difference to the joint spatio-temporally predicted block to obtain the block.
2. The method as recited in claim 1 , wherein the joint spatio-temporally predicted block comprises a coding mode of an H.264/AVC video coding standard.
3. The method as recited in claim 1 , wherein applying the subband decomposition for decomposing at least one of the inter frame prediction or the intra frame prediction includes:
applying a 1-dimensional low-pass filter horizontally and vertically to the block to obtain the low-pass component; and
subtracting the low-pass component from the block to obtain the high-pass component.
4. The method as recited in claim 1 , wherein applying the subband decomposition for decomposing at least one of the inter frame prediction or the intra frame prediction includes:
applying a 2-dimensional low-pass filter to the block to obtain the low-pass component; and
subtracting the low-pass component from the block to obtain the high-pass component.
5. A system, comprising:
one or more hardware processors of a computing device configured to receive a bitstream representing a video sequence, the bitstream including a motion vector, a direction vector, and a difference that is to be added to a joint spatio-temporally predicted block to obtain a block of a frame in the video sequence; and
a decoder, implemented by the one or more hardware processors, to determine, at least partly in response to receiving the bitstream, the joint spatio-temporally predicted block, the decoder configured to:
determine an inter frame prediction of the block based at least in part on the motion vector;
decompose, applying a subband decomposition, the inter frame prediction into a high-pass component and a low-pass component;
determine an intra frame prediction of the block based at least in part on the direction vector;
decompose, applying a subband decomposition, the intra frame prediction into a high-pass component and a low-pass component; and
combine the low-pass component of the intra frame prediction with the high-pass component of the inter frame prediction to obtain the joint spatio-temporally predicted block; and
wherein the one or more hardware processors of the computing device are further configured to add the difference to the joint spatio-temporally predicted block to obtain the block.
6. The system as recited in claim 5 , wherein the joint spatio-temporally predicted block comprises a coding mode of an H.264/AVC video coding standard.
7. The system as recited in claim 5 , further comprising a subband decomposition engine that includes a low-pass filter selected from the group of filters consisting of:
a 1-dimensional low-pass filter configured to be applied horizontally and vertically to the block to obtain the low-pass component of at least one of the inter frame prediction or the intra frame prediction; and
a 2-dimensional low-pass filter configured to be applied to the block to obtain the low-pass component of at least one of the inter frame prediction or the intra frame prediction,
wherein the subband decomposition engine subtracts the low-pass component from the block to obtain the high-pass component.