Systems, apparatus and methods for inter prediction refinement with optical flow
Method, apparatus and systems are disclosed. In one embodiment, a method of decoding includes obtaining a sub-block based motion prediction signal for a current block of the video; obtaining one or more spatial gradients of the sub-block based motion prediction signal or one or more motion vector difference values; obtaining a refinement signal for the current block based on the one or more obtained spatial gradients or the one or more obtained motion vector difference values; obtaining a refined motion prediction signal for the current block based on the sub-block based motion prediction signal and the refinement signal; and decoding the current block based on the refined motion prediction signal.
1 . A method of decoding a video, comprising:
generating a sub-block-based motion prediction signal for a sub-block of a block of a picture based on an affine motion model associated with said block;
determining a set of pixel-level motion vector difference values for said sub-block using said affine motion model associated with said block;
determining spatial gradients of said sub-block-based motion prediction signal, for each sample location of said sub-block, wherein an extended sub-block is formed to include said sub-block-based motion prediction signal and a plurality of samples surrounding said sub-block, and wherein each of said plurality of samples surrounding said sub-block is obtained based on integer motion compensation;
determining a motion prediction refinement signal for said sub-block, based on said determined set of pixel-level motion vector difference values and said determined spatial gradients;
combining said motion prediction signal and said motion prediction refinement signal to produce a refined motion prediction signal for said sub-block; and
decoding the video using the refined motion prediction signal.
2 . The method of claim 1 , wherein said integer motion compensation is based on an integer part of a motion vector of said sub-block.
3 . The method of claim 1 , wherein said integer motion compensation is based on a nearest integer motion vector of a motion vector of said sub-block.
4 . The method of claim 1 , wherein said sub-block is extended by one sample in each direction to form said extended sub-block.
5 . A computer readable storage medium having stored thereon instructions for decoding video data according to the method of claim 1 .
6 . A method of encoding a video, comprising:
generating a sub-block-based motion prediction signal for a sub-block of a block of a picture based on an affine motion model associated with said block;
determining a set of pixel-level motion vector difference values for said sub-block using said affine motion model associated with said block;
determining spatial gradients of said sub-block-based motion prediction signal, for each sample location of said sub-block, wherein an extended sub-block is formed to include said sub-block-based motion prediction signal and a plurality of samples surrounding said sub-block, and wherein each of said plurality of samples surrounding said sub-block is obtained based on integer motion compensation;
determining a motion prediction refinement signal for said sub-block, based on said determined set of pixel-level motion vector difference values and said determined spatial gradients;
combining said motion prediction signal and said motion prediction refinement signal to produce a refined motion prediction signal for said sub-block; and
encoding the video using the refined motion prediction signal.
7 . The method of claim 6 , wherein said integer motion compensation is based on an integer part of a motion vector of said sub-block.
8 . The method of claim 6 , wherein said integer motion compensation is based on a nearest integer motion vector of a motion vector of said sub-block.
9 . The method of claim 6 , wherein said sub-block is extended by one sample in each direction to form said extended sub-block.
10 . A computer readable storage medium having stored thereon instructions for encoding video data according to the method of claim 6 .
11 . An apparatus for decoding a video, comprising a processor configured to:
generate a sub-block-based motion prediction signal for a sub-block of a block of a picture based on an affine motion model associated with said block;
determine a set of pixel-level motion vector difference values for said sub-block using said affine motion model associated with said block;
determine spatial gradients of said sub-block-based motion prediction signal, for each sample location of said sub-block, wherein an extended sub-block is formed to include said sub-block-based motion prediction signal and a plurality of samples surrounding said sub-block, and wherein each of said plurality of samples surrounding said sub-block is obtained based on integer motion compensation;
determine a motion prediction refinement signal for said sub-block, based on said determined set of pixel-level motion vector difference values and said determined spatial gradients;
combine said motion prediction signal and said motion prediction refinement signal to produce a refined motion prediction signal for said sub-block; and
decode the video using the refined motion prediction signal.
12 . The apparatus of claim 11 , wherein said integer motion compensation is based on an integer part of a motion vector of said sub-block.
13 . The apparatus of claim 11 , wherein said integer motion compensation is based on a nearest integer motion vector of a motion vector of said sub-block.
14 . The apparatus of claim 11 , wherein said sub-block is extended by one sample in each direction to form said extended sub-block.
15 . An apparatus for encoding a video, comprising a processor configured to:
generate a sub-block-based motion prediction signal for a sub-block of a block of a picture based on an affine motion model associated with said block;
determine a set of pixel-level motion vector difference values for said sub-block using said affine motion model associated with said block;
determine spatial gradients of said sub-block-based motion prediction signal, for each sample location of said sub-block, wherein an extended sub-block is formed to include said sub-block-based motion prediction signal and a plurality of samples surrounding said sub-block, and wherein each of said plurality of samples surrounding said sub-block is obtained based on integer motion compensation;
determine a motion prediction refinement signal for said sub-block, based on said determined set of pixel-level motion vector difference values and said determined spatial gradients;
combine said motion prediction signal and said motion prediction refinement signal to produce a refined motion prediction signal for said sub-block; and
encode the video using the refined motion prediction signal.
16 . The apparatus of claim 15 , wherein said integer motion compensation is based on an integer part of a motion vector of said sub-block.
17 . The apparatus of claim 15 , wherein said integer motion compensation is based on a nearest integer motion vector of a motion vector of said sub-block.
18 . The apparatus of claim 15 , wherein said sub-block is extended by one sample in each direction to form said extended sub-block.