Methods and apparatus for prediction refinement for decoder side motion vector refinement with optical flow
Methods, devices, apparatus, systems, architectures and interfaces to improve motion vector (MV) refinement based sub-block (SB) level motion compensated prediction are provided. A decoding method includes receiving a bitstream of encoded video data, the bitstream including at least one block of video data including a plurality of SBs; performing a MV derivation, including a decoder based MV (DMVR) process, for at least one SB in the block to generate a refined MV for each SB; performing SB based motion compensation on the at least one sub-block to generate a SB based prediction within each SB; obtaining a spatial gradient for the prediction within each SB; determining a MV offset for each pixel in each SB; obtaining an intensity change in each SB based on the spatial gradients and MV offsets via an optical flow equation; and refining the prediction within each SB based on the obtained intensity changes.
1. A method for video decoding, comprising:
receiving a block of video data, wherein the block of video data includes a plurality of sub-blocks; and
for a sub-block of the plurality of sub-blocks, refining a prediction of the sub-block by:
deriving motion vectors, each indicating a refinement motion of a prediction of a respective one of the plurality of sub-blocks,
computing an affine motion model based on the derived motion vectors,
determining, for pixels of the prediction of the sub-block, respective motion vector offsets using the affine motion model,
obtaining an intensity change for the pixels based on respective spatial gradients and the respective motion vector offsets via an optical flow equation, and
refining the pixels based on the obtained intensity change.
2. The method of claim 1 , wherein the derived motion vectors are obtained by a decoder-based motion vector refinement process.
3. The method of claim 1 , wherein a model error is estimated based on the derived motion vectors, and if the estimated model error exceeds a threshold the computing, the determining, the obtaining, and the refining for the sub-block are skipped.
4. The method of claim 1 , wherein the plurality of sub-blocks includes sub-blocks neighboring the sub-block and wherein the computing of the affine motion model is based on motion vectors derived for the neighboring sub-blocks, and further comprising:
estimating a model error of the affine motion model using the motion vector derived for the sub-block; and
if the estimated model error exceeds a threshold, skipping the determining, the obtaining, and the refining for the sub-block.
5. The method of claim 1 , wherein the affine motion model computed for the sub-block is used for refining a prediction of another sub-block of the video block.
6. The method of claim 1 , wherein the affine motion model is a four-parameter affine motion model for one or more sub-blocks of the plurality of sub-blocks, and wherein the one or more sub-blocks are located at a left boundary and/or a top boundary of the block of video data.
7. The method of claim 1 , wherein the computing of the affine motion model further comprises:
computing multiple models based on the derived motion vectors;
estimating respective model errors for the multiple models; and
selecting as the affine motion model one of the multiple models having the least model error.
8. The method of claim 1 , wherein the computing of the affine motion model further comprises:
computing multiple models based on the derived motion vectors, wherein the multiple models include one or more of a six-parameter affine motion model and a four-parameter affine motion model.
9. The method of claim 1 , wherein the refining the pixels based on the obtained intensity change comprises:
weighting the obtained intensity change by a weighting factor, and
adding the weighted intensity change to the pixels.
10. The method of claim 9 , further comprising:
receiving a weight index indicating, at the video block level or at a picture level, the weighting factor for weighting the obtained intensity change.
11. A method for video encoding, comprising:
receiving a block of video data, wherein the block of video data includes a plurality of sub-blocks; and
for a sub-block of the plurality of sub-blocks, refining a prediction of the sub-block by:
deriving motion vectors, each indicating a refinement motion of a prediction of a respective one of the plurality of sub-blocks,
computing an affine motion model based on the derived motion vectors,
determining, for pixels of the prediction of the sub-block, respective motion vector offsets using the affine motion model,
obtaining an intensity change for the pixels based on respective spatial gradients and the respective motion vector offsets via an optical flow equation, and
refining the pixels based on the obtained intensity change.
12. An apparatus for video decoding, comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the apparatus to:
receive a block of video data, wherein the block of video data includes a plurality of sub-blocks, and
for a sub-block of the plurality of sub-blocks, refine a prediction of the sub-block by:
deriving motion vectors, each indicating a refinement motion of a prediction of a respective one of the plurality of sub-blocks,
computing an affine motion model based on the derived motion vectors,
determining, for pixels of the prediction of the sub-block, respective motion vector offsets using the affine motion model,
obtaining an intensity change for the pixels based on respective spatial gradients and the respective motion vector offsets via an optical flow equation, and
refining the pixels based on the obtained intensity change.
13. The apparatus of claim 12 , wherein the derived motion vectors are obtained by a decoder-based motion vector refinement process.
14. The apparatus of claim 12 , wherein the plurality of sub-blocks includes sub-blocks neighboring the sub-block, wherein the computing of the affine motion model is based on motion vectors derived for the neighboring sub-blocks, and wherein the instructions further cause the apparatus to:
estimate a model error of the affine motion model using the motion vector derived for the sub-block; and
if the estimated model error exceeds a threshold, skip the determining, the obtaining, and the refining for the sub-block.
15. The apparatus of claim 12 , wherein the affine motion model computed for the sub-block is used for refining a prediction of another sub-block of the video block.
16. The apparatus of claim 12 , wherein the computing of the affine motion model further comprises:
computing multiple models based on the derived motion vectors;
estimating respective model errors for the multiple models; and
selecting as the affine motion model one of the multiple models having the least model error.
17. The apparatus of claim 12 , wherein the computing of the affine motion model further comprises:
computing multiple models based on the derived motion vectors, wherein the multiple models include one or more of a six-parameter affine motion model and a four-parameter affine motion model.
18. The apparatus of claim 12 , wherein the refining the pixels based on the obtained intensity change comprises:
weighting the obtained intensity change by a weighting factor, and
adding the weighted intensity change to the pixels.
19. The apparatus of claim 18 , further comprising:
receiving a weight index indicating, at the video block level or at a picture level, the weighting factor for weighting the obtained intensity change.
20. An apparatus for video encoding, comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the apparatus to:
receive a block of video data, wherein the block of video data includes a plurality of sub-blocks, and
for a sub-block of the plurality of sub-blocks, refine a prediction of the sub-block by:
deriving motion vectors, each indicating a refinement motion of a prediction of a respective one of the plurality of sub-blocks,
computing an affine motion model based on the derived motion vectors,
determining, for pixels of the prediction of the sub-block, respective motion vector offsets using the affine motion model,
obtaining an intensity change for the pixels based on respective spatial gradients and the respective motion vector offsets via an optical flow equation, and
refining the pixels based on the obtained intensity change.