Bi-prediction refinement in affine with optical flow
PROF (Prediction Refinement with Optical Flow) and BDOF (Bi-directional Optical Flow) use the optical flow model to provide sample-wise motion compensation refinement. In one implementation, to calculate the spatial gradients in BDOF, an extended region is generated around a block to be coded. To simplify the computation, BDOF may use the same padding operations as in PROF, namely, a padding sample is copied from the nearest integer neighbor in the reference picture. BDOF and PROF can also be used together when sub-block based affine motion-compensated prediction is used. BDOF can be applied after PROF generates new prediction blocks, or BDOF can use the spatial gradients in the original prediction blocks to re-use the computations with PROF. In addition, BDOF and PROF operations can be combined, and the prediction adjustment will be directly based on the spatial gradients in the original prediction blocks.
1. A method, comprising:
accessing a block to be decoded in affine mode, said block including a plurality of sub-blocks;
for one sub-block of said plurality of sub-blocks, obtaining a first and second prediction block, from a first and second reference picture, respectively, using sub-block based affine motion-compensated prediction, said one block in a bi-prediction mode;
obtaining motion refinement for said one sub-block, based on said first and second prediction blocks and spatial gradients in said first and second prediction blocks, according to a temporal optical flow model and a spatial optical flow model, wherein said motion refinement for said one sub-block is based on a linear combination of a motion difference according to said spatial optical flow model and an initial motion refinement for said one sub-block according to said temporal optical flow model;
obtaining a prediction adjustment for said one sub-block, based on said motion refinement and said spatial gradients; and
decoding said one sub-block based on said prediction adjustment, said first prediction block and said second prediction block.
2. The method of claim 1 , wherein said obtaining a prediction adjustment comprises:
scaling a horizontal gradient and a vertical gradient for said first prediction block by a horizontal motion refinement and a vertical motion refinement for said first prediction block, respectively; and
scaling a horizontal gradient and a vertical gradient for said second prediction block by a horizontal motion refinement and a vertical motion refinement for said second prediction block, respectively.
3. The method of claim 2 , further comprising:
obtaining a motion difference between a motion vector at a sample level and a motion vector for said first prediction block, for each sample in said first prediction block,
wherein said horizontal motion refinement and said vertical motion refinement for said first prediction block are obtained based on said motion difference.
4. The method of claim 1 , wherein said first and second prediction blocks and spatial gradients in said first and second prediction blocks are directly used for said motion refinement and said prediction adjustment for said one sub-block.
5. An apparatus, comprising one or more processors, wherein said one or more processors are configured to:
access a block to be decoded in affine mode, said block including a plurality of sub-blocks;
for one sub-block of said plurality of sub-blocks, obtain a first and second prediction block, from a first and second reference picture, respectively, using sub-block based affine motion-compensated prediction, said one block in a bi-prediction mode;
obtain motion refinement for said one sub-block, based on said first and second prediction blocks and spatial gradients in said first and second prediction blocks, according to a temporal optical flow model and a spatial optical flow model, wherein said motion refinement for said one sub-block is based on a linear combination of a motion difference according to said spatial optical flow model and an initial motion refinement for said one sub-block according to said temporal optical flow model;
obtain a prediction adjustment for said one sub-block, based on said motion refinement and said spatial gradients; and
decode said one sub-block based on said prediction adjustment, said first prediction block and said second prediction block.
6. The apparatus of claim 5 , wherein said one or more processors are further configured to:
scale a horizontal gradient and a vertical gradient for said first prediction block by a horizontal motion refinement and a vertical motion refinement for said first prediction block, respectively; and
scale a horizontal gradient and a vertical gradient for said second prediction block by a horizontal motion refinement and a vertical motion refinement for said second prediction block, respectively.
7. The apparatus of claim 6 , wherein said one or more processors are further configured to:
obtain a motion difference between a motion vector at a sample level and a motion vector for said first prediction block, for each sample in said first prediction block,
wherein said horizontal motion refinement and said vertical motion refinement for said first prediction block are obtained based on said motion difference.
8. The apparatus of claim 5 , wherein said first and second prediction blocks and spatial gradients in said first and second prediction blocks are directly used for said motion refinement and said prediction adjustment for said one sub-block.
9. A method, comprising:
accessing a block to be encoded in affine mode, said block including a plurality of sub-blocks;
for one sub-block of said plurality of sub-blocks, obtaining a first and second prediction block, from a first and second reference picture, respectively, using sub-block based affine motion-compensated prediction, said one block in a bi-prediction mode;
obtaining motion refinement for said one sub-block, based on said first and second prediction blocks and spatial gradients in said first and second prediction blocks, according to a temporal optical flow model and a spatial optical flow model, wherein said motion refinement for said one sub-block is based on a linear combination of a motion difference according to said spatial optical flow model and an initial motion refinement for said one sub-block according to said temporal optical flow model;
obtaining a prediction adjustment for said one sub-block, based on said motion refinement and said spatial gradients; and
encoding said one sub-block responsive to said prediction adjustment, said first prediction block and said second prediction block.
10. The method of claim 9 , wherein said obtaining a prediction adjustment comprises:
scaling a horizontal gradient and a vertical gradient for said first prediction block by a horizontal motion refinement and a vertical motion refinement for said first prediction block, respectively; and
scaling a horizontal gradient and a vertical gradient for said second prediction block by a horizontal motion refinement and a vertical motion refinement for said second prediction block, respectively.
11. The method of claim 10 , further comprising:
obtaining a motion difference between a motion vector at a sample level and a motion vector for said first prediction block, for each sample in said first prediction block,
wherein said horizontal motion refinement and said vertical motion refinement for said first prediction block is obtained based on said motion difference.
12. The method of claim 9 , wherein said first and second prediction blocks and spatial gradients in said first and second prediction blocks are directly used for said motion refinement and said prediction adjustment for said one sub-block.
13. An apparatus, comprising one or more processors, wherein said one or more processors are configured to:
access a block to be encoded in affine mode, said block including a plurality of sub-blocks;
for one sub-block of said plurality of sub-blocks, obtain a first and second prediction block, from a first and second reference picture, respectively, using sub-block based affine motion-compensated prediction, said one block in a bi-prediction mode;
obtain motion refinement for said one sub-block, based on said first and second prediction blocks and spatial gradients in said first and second prediction blocks, according to a temporal optical flow model and a spatial optical flow model, wherein said motion refinement for said one sub-block is based on a linear combination of a motion difference according to said spatial optical flow model and an initial motion refinement for said one sub-block according to said temporal optical flow model;
obtain a prediction adjustment for said one sub-block, based on said motion refinement and said spatial gradients; and
encode said one sub-block responsive to said prediction adjustment, said first prediction block and said second prediction block.
14. The apparatus of claim 13 , wherein said one or more processors are further configured to:
scale a horizontal gradient and a vertical gradient for said first prediction block by a horizontal motion refinement and a vertical motion refinement for said first prediction block, respectively; and
scale a horizontal gradient and a vertical gradient for said second prediction block by a horizontal motion refinement and a vertical motion refinement for said second prediction block, respectively.
15. The apparatus of claim 14 , wherein said one or more processors are further configured to:
obtain a motion difference between a motion vector at a sample level and a motion vector for said first prediction block, for each sample in said first prediction block,
wherein said horizontal motion refinement and said vertical motion refinement for said first prediction block is obtained based on said motion difference.
16. The apparatus of claim 13 , wherein said first and second prediction blocks and spatial gradients in said first and second prediction blocks are directly used for said motion refinement and said prediction adjustment for said one sub-block.