Affine inter prediction refinement with optical flow
An apparatus for video decoding includes processing circuitry. The processing circuitry can be configured to determine an intermediate motion vector MV block of a current block having a first precision that is higher than a second precision supported by an interpolation filter. The processing circuitry can be configured to round the intermediate motion vector MV block to generate a rounded motion vector MV round of the current block having the second precision supported by the interpolation filter. The processing circuitry can be configured to generate a prediction block of the current block based on the rounded motion vector MV round and the interpolation filter. The processing circuitry can be configured to determine a prediction refinement offset for a prediction sample of the prediction block, and add the prediction refinement offset to the prediction sample of the prediction block to generate a refined prediction sample.
1. A method of video decoding by a video decoder, comprising:
determining an intermediate motion vector MV block of a current block having a first precision that is higher than a second precision supported by an interpolation filter;
rounding the intermediate motion vector MV block to generate a rounded motion vector MV round of the current block having the second precision supported by the interpolation filter;
generating a prediction block of the current block based on the rounded motion vector MV round and the interpolation filter;
determining a prediction refinement offset for a prediction sample of the prediction block based on gradients of the prediction sample of the prediction block and motion vector differences between the intermediate motion vector MV block and the rounded motion vector MV round ; and
adding the prediction refinement offset to the prediction sample of the prediction block to generate a refined prediction sample.
2. The method of claim 1 , wherein the gradients of the prediction sample of the prediction block are calculated with a 3-tap filter [−1, 0, 1].
3. The method of claim 1 , wherein the determining the prediction refinement offset comprises:
determining a prediction refinement offset ΔI (i,j) according to:
Δ I ( i,j )= g x ( i,j )*Δ v x +g y ( i,j )*Δ v y ,
where g x (i,j) and g y (i,j) are gradients of the prediction sample of the prediction block at a location (i,j) along x and y directions, respectively, and Δv x , and Δv y are motion vector differences between the intermediate motion vector MV block and the rounded motion vector MV round along x and y directions, respectively.
4. The method of claim 1 , wherein the current block is a block or a sub-block coded with a temporal motion vector prediction mode (TMVP), a sub-block-based TMVP mode (SbTMVP), merge with motion vector difference (MMVD), affine advance motion vector prediction (AMVP), or a spatial motion vector prediction mode where the intermediate motion vector MV block is generated in a motion vector scaling operation.
5. The method of claim 1 , wherein the current block is coded using a motion vector predictor with the first precision.
6. The method of claim 1 , further comprising:
receiving a syntax element indicating the first precision in a sequence parameter set (SPS), picture parameter set (PPS), a slice header, a tile group header.
7. The method of claim 1 , further comprising:
storing the intermediate motion vector MV block having the first precision in a memory; and
coding another block in a current picture including the current block using the intermediate motion vector MV block for spatial motion vector prediction.
8. A method of video encoding by a video encoder, comprising:
determining an intermediate motion vector MV block of a current block having a first precision that is higher than a second precision supported by an interpolation filter;
rounding the intermediate motion vector MV block to generate a rounded motion vector MV round of the current block having the second precision supported by the interpolation filter;
obtaining a prediction block of the current block based on the rounded motion vector MV round and the interpolation filter;
determining a prediction refinement offset for a prediction sample of the prediction block based on gradients of the prediction sample of the prediction block and motion vector differences between the intermediate motion vector MV block and the rounded motion vector MV round ; and
adding the prediction refinement offset to the prediction sample of the prediction block to generate a refined prediction sample.
9. The method of claim 8 , wherein the gradients of the prediction sample of the prediction block are calculated with a 3-tap filter [−1, 0, 1].
10. The method of claim 8 , wherein the determining the prediction refinement offset comprises:
determining a prediction refinement offset ΔI (i,j) according to:
Δ I ( i,j )= g x ( i,j )*Δ v x +g y ( i,j )*Δ v y ,
where g x (i,j) and g y (i,j) are gradients of the prediction sample of the prediction block at a location (i,j) along x and y directions, respectively, and Δv x , and Δv y are motion vector differences between the intermediate motion vector MV block and the rounded motion vector MV round along x and y directions, respectively.
11. The method of claim 8 , wherein the current block is a block or a sub-block to be coded with a temporal motion vector prediction mode (TMVP), a sub-block-based TMVP mode (SbTMVP), merge with motion vector difference (MMVD), affine advance motion vector prediction (AMVP), or a spatial motion vector prediction mode where the intermediate motion vector MV block is generated in a motion vector scaling operation.
12. The method of claim 8 , further comprising:
encoding the current block in a bitstream using a motion vector predictor with the first precision.
13. The method of claim 8 , further comprising:
encoding a syntax element indicating the first precision in a sequence parameter set (SPS), picture parameter set (PPS), a slice header, a tile group header in a bitstream.
14. The method of claim 8 , further comprising:
storing the intermediate motion vector MV block having the first precision in a memory; and
encoding another block in a current picture including the current block using the intermediate motion vector MV block for spatial motion vector prediction.
15. A method of processing visual media data, the method comprising:
processing a bitstream that includes the visual media data according to a format rule, wherein
the bitstream includes prediction information of a current block; and
the format rule specifies that
when an intermediate motion vector MV block of the current block has a first precision that is higher than a second precision supported by an interpolation filter, the intermediate motion vector MV block is rounded to generate a rounded motion vector MV round of the current block having the second precision supported by the interpolation filter,
a prediction block of the current block is generated based on the rounded motion vector MV round and the interpolation filter,
a prediction refinement offset for a prediction sample of the prediction block is determined based on gradients of the prediction sample of the prediction block and motion vector differences between the intermediate motion vector MV block and the rounded motion vector MV round , and
the prediction refinement offset is added to the prediction sample of the prediction block to generate a refined prediction sample.
16. The method of claim 15 , wherein the gradients of the prediction sample of the prediction block are calculated with a 3-tap filter [−1, 0, 1].
17. The method of claim 15 , wherein the format rule specifies that:
a prediction refinement offset ΔI (i,j) is determined according to:
Δ I ( i,j )= g x ( i,j )*Δ v x +g y ( i,j )*Δ v y ,
where g x (i,j) and g y (i,j) are gradients of the prediction sample of the prediction block at a location (i,j) along x and y directions, respectively, and Δv x , and Δv y are motion vector differences between the intermediate motion vector MV block and the rounded motion vector MV round along x and y directions, respectively.
18. The method of claim 15 , wherein the current block is a block or a sub-block coded with a temporal motion vector prediction mode (TMVP), a sub-block-based TMVP mode (SbTMVP), merge with motion vector difference (MMVD), affine advance motion vector prediction (AMVP), or a spatial motion vector prediction mode where the intermediate motion vector MV block is generated in a motion vector scaling operation.
19. The method of claim 15 , wherein the current block is coded using a motion vector predictor with the first precision.
20. The method of claim 15 , wherein the bitstream includes a syntax element indicating the first precision in a sequence parameter set (SPS), picture parameter set (PPS), a slice header, a tile group header.