Systems, apparatus and methods for inter prediction refinement with optical flow
Method, apparatus and systems are disclosed. In one embodiment, a method of decoding includes obtaining a sub-block based motion prediction signal for a current block of the video; obtaining one or more spatial gradients of the sub-block based motion prediction signal or one or more motion vector difference values; obtaining a refinement signal for the current block based on the one or more obtained spatial gradients or the one or more obtained motion vector difference values; obtaining a refined motion prediction signal for the current block based on the sub-block based motion prediction signal and the refinement signal; and decoding the current block based on the refined motion prediction signal.
1. A method of decoding a video, the method comprising:
generating a sub-block-based motion prediction signal for a sub-block of a block of a picture based on an affine motion model associated with said block;
determining a set of pixel-level motion vector difference values for said sub-block using said affine motion model associated with said block, wherein a pixel-level motion vector difference for a sample is based on a difference between a location of said sample and a center position of said sub-block;
determining spatial gradients of said sub-block-based motion prediction signal, for each sample location of said sub-block;
determining a motion prediction refinement signal for said sub-block, based on said determined set of pixel-level motion vector difference values and said determined spatial gradients;
combining said motion prediction signal and said motion prediction refinement signal to produce a refined motion prediction signal for said sub-block; and
decoding the video using the refined motion prediction signal.
2. The method of claim 1 , wherein said motion prediction refinement signal is determined based on optical flow.
3. The method of claim 1 , wherein said motion prediction refinement signal for said sub-block is clipped before being combined with said motion prediction signal to produce said refined motion prediction signal.
4. The method of claim 3 , wherein said clipping is based on a bit depth of prediction samples of said block.
5. A computer readable storage medium having stored thereon instructions for decoding video data according to the method of claim 1 .
6. A method of encoding a video, the method comprising:
generating a sub-block-based motion prediction signal for a sub-block of a block of a picture based on an affine motion model associated with said block;
determining a set of pixel-level motion vector difference values for said sub-block using said affine motion model associated with said block, wherein a pixel-level motion vector difference for a sample is based on a difference between a location of said sample and a center position of said sub-block;
determining spatial gradients of said sub-block-based motion prediction signal, for each sample location of said sub-block;
determining a motion prediction refinement signal for said sub-block, based on said determined set of pixel-level motion vector difference values and said determined spatial gradients;
combining said motion prediction signal and said motion prediction refinement signal to produce a refined motion prediction signal for said sub-block; and
encoding the video using the refined motion prediction signal.
7. The method of claim 6 , wherein said motion prediction refinement signal is determined based on optical flow.
8. The method of claim 6 , wherein said motion prediction refinement signal for said sub-block is clipped before being combined with said motion prediction signal to produce said refined motion prediction signal.
9. The method of claim 8 , wherein said clipping is based on a bit depth of prediction samples of said block.
10. A computer readable storage medium having stored thereon instructions for decoding video data according to the method of claim 6 .
11. An apparatus for decoding a video, comprising a processor configured to:
generate a sub-block-based motion prediction signal for a sub-block of a block of a picture based on an affine motion model associated with said block;
determine a set of pixel-level motion vector difference values for said sub-block using said affine motion model associated with said block, wherein a pixel-level motion vector difference for a sample is based on a difference between a location of said sample and a center position of said sub-block;
determine spatial gradients of said sub-block-based motion prediction signal, for each sample location of said sub-block;
determine a motion prediction refinement signal for said sub-block, based on said determined set of pixel-level motion vector difference values and said determined spatial gradients;
combine said motion prediction signal and said motion prediction refinement signal to produce a refined motion prediction signal for said sub-block; and
decode the video using the refined motion prediction signal.
12. The apparatus of claim 11 , wherein said motion prediction refinement signal is determined based on optical flow.
13. The apparatus of claim 11 , wherein said motion prediction refinement signal for said sub-block is clipped before being combined with said motion prediction signal to produce said refined motion prediction signal.
14. The apparatus of claim 13 , wherein said clipping is based on a bit depth of prediction samples of said block.
15. The apparatus of claim 11 , wherein a precision of said spatial gradients is kept at a same precision as said sub-block-based motion prediction signal for said sub-block.
16. An apparatus for encoding a video, comprising a processor configured to:
generate a sub-block-based motion prediction signal for a sub-block of a block of a picture based on an affine motion model associated with said block;
determine a set of pixel-level motion vector difference values for said sub-block using said affine motion model associated with said block, wherein a pixel-level motion vector difference for a sample is based on a difference between a location of said sample and a center position of said sub-block;
determine spatial gradients of said sub-block-based motion prediction signal, for each sample location of said sub-block;
determine a motion prediction refinement signal for said sub-block, based on said determined set of pixel-level motion vector difference values and said determined spatial gradients;
combine said motion prediction signal and said motion prediction refinement signal to produce a refined motion prediction signal for said sub-block; and
encode the video using the refined motion prediction signal.
17. The apparatus of claim 16 , wherein said motion prediction refinement signal is determined based on optical flow.
18. The apparatus of claim 16 , wherein said motion prediction refinement signal for said sub-block is clipped before being combined with said motion prediction signal to produce said refined motion prediction signal.
19. The apparatus of claim 18 , wherein said clipping is based on a bit depth of prediction samples of said block.
20. The apparatus of claim 16 , wherein a precision of said spatial gradients is kept at a same precision as said sub-block-based motion prediction signal for said sub-block.