Affine inter prediction refinement with optical flow

1. A method of video coding by a video encoder, comprising:

determining a picture level syntax element indicating whether a prediction refinement with optical flow (PROF) for affine prediction is disabled for a current picture;

determining whether to apply the PROF for affine prediction to an affine coded block in the current picture based on the picture level syntax element indicating whether the PROF for affine prediction is disabled for the current picture,

wherein responsive to the picture level syntax element indicating the PROF for affine prediction is disabled for the current picture, the PROF for affine prediction is not applied to the affine coded block in the current picture; and

responsive to the PROF for affine prediction being determined as applied to the affine coded block:

generating spatial gradients g x (i,j) and g y (i,j) at a sample location (i,j) in the affine coded block in a prediction sample I(i,j);

generating a prediction refinement ΔI(i,j) based on the spatial gradients g x (i,j) and g y (i,j); and

adding the prediction refinement ΔI(i,j) to the prediction sample I(i,j) to generate a refined prediction sample.

2. The method of claim 1 , wherein responsive to the picture level syntax element indicating the PROF for affine prediction is not disabled for the current picture, the PROF for affine prediction is applied to the affine coded block.

3. The method of claim 1 , wherein the PROF for affine prediction is enabled by default.

4. The method of claim 1 , wherein the determining whether to apply the PROF comprises:

determining whether to apply the PROF for affine prediction to the affine coded block based on values of affine parameters of an affine model of the affine coded block.

5. The method of claim 1 , wherein the generating the spatial gradients comprises:

generating the spatial gradients g x (i,j) and g y (i,j) at the sample location (i,j) based on a first prediction sample(s) of a first sub-block including the prediction sample I(i,j) and a second prediction sample(s) of a second sub-block neighboring the first sub-block, the first sub-block and the second sub-block being partitioned from the affine coded block.

6. The method of claim 1 , wherein the generating the spatial gradients comprises:

performing inter prediction for sub-blocks of the affine coded block; and

generating spatial gradients at sample locations on a basis of prediction samples of the entire affine coded block.

7. The method of claim 1 , wherein the generating the spatial gradients comprises:

generating the spatial gradients g x (i,j) and g y (i,j) at the sample location (i,j) using a generated gradient filter on reference samples in a reference picture of the affine coded block.

8. The method of claim 7 , wherein

the generated gradient filter is generated by a convolution of a first gradient filter and an interpolation filter,

application of the interpolation filter on the reference samples in the reference picture of the affine coded block generates prediction samples of the affine coded block, and

subsequent application of the first gradient filter on the generated prediction samples of the affine coded block generates the spatial gradients g x (i,j) and g y (i,j).

9. The method of claim 1 , wherein the determining whether to apply the PROF comprises:

determining whether to apply the PROF for affine prediction to the affine coded block based on the picture level syntax element indicating whether the PROF for affine prediction is disabled and a syntax element indicating whether a local illumination compensation (LIC) is enabled.

10. The method of claim 1 , wherein the affine coded block is coded in an affine merge mode, and an LIC flag value of the affine coded block is inherited from a neighboring block of the affine coded block that is used as a source for affine model inheritance or affine model construction of the affine coded block.

11. An apparatus, comprising circuitry configured to:

determine a picture level syntax element indicating whether a prediction refinement with optical flow (PROF) for affine prediction is disabled for a current picture;

determine whether to apply the PROF for affine prediction to an affine coded block in the current picture based on the picture level syntax element indicating whether the PROF for affine prediction is disabled for the current picture,

wherein responsive to the picture level syntax element indicating the PROF for affine prediction is disabled for the current picture, the PROF for affine prediction is not applied to the affine coded block in the current picture; and

responsive to the PROF for affine prediction being determined as applied to the affine coded block:

generate spatial gradients g x (i,j) and g y (i,j) at a sample location (i,j) in the affine coded block in a prediction sample I(i,j);

generate a prediction refinement ΔI(i,j) based on the spatial gradients g x (i,j) and g y (i,j); and

add the prediction refinement ΔI(i,j) to the prediction sample I(i,j) to generate a refined prediction sample.

12. The apparatus of claim 11 , wherein responsive to the picture level syntax element indicating the PROF for affine prediction is not disabled for the current picture, the PROF for affine prediction is applied to the affine coded block.

13. The apparatus of claim 11 , wherein the PROF for affine prediction is enabled by default.

14. The apparatus of claim 11 , wherein the circuitry is further configured to:

determine whether to apply the PROF for affine prediction to the affine coded block based on values of affine parameters of an affine model of the affine coded block.

15. The apparatus of claim 11 , wherein the circuitry is further configured to:

generate the spatial gradients g x (i,j) and g y (i,j) at the sample location (i,j) based on a first prediction sample(s) of a first sub-block including the prediction sample I(i,j) and a second prediction sample(s) of a second sub-block neighboring the first sub-block, the first sub-block and the second sub-block being partitioned from the affine coded block.

16. The apparatus of claim 11 , wherein the circuitry is further configured to:

perform inter prediction for sub-blocks of the affine coded block; and

generate spatial gradients at sample locations on a basis of prediction samples of the entire affine coded block.

17. The apparatus of claim 11 , wherein the circuitry is further configured to:

generate the spatial gradients g x (i,j) and g y (i,j) at the sample location (i,j) using a generated gradient filter on reference samples in a reference picture of the affine coded block.

18. The apparatus of claim 17 , wherein the generated gradient filter is generated by a convolution of a first gradient filter and an interpolation filter, application of the interpolation filter on the reference samples in the reference picture of the affine coded block generates prediction samples of the affine coded block, and subsequent application of the first gradient filter on the generated prediction samples of the affine coded block generates the spatial gradients g x (i,j) and g y (i,j).

19. The apparatus of claim 11 , wherein the circuitry is further configured to:

determine whether to apply the PROF for affine prediction to the affine coded block based on the picture level syntax element indicating whether the PROF for affine prediction is disabled and a syntax element indicating whether a local illumination compensation (LIC) is enabled.

20. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method of video coding, the method comprises:

determining a picture level syntax element indicating whether a prediction refinement with optical flow (PROF) for affine prediction is disabled for a current picture;

determining whether to apply the PROF for affine prediction to an affine coded block in the current picture based on the picture level syntax element indicating whether the PROF for affine prediction is disabled for the current picture,

wherein responsive to the picture level syntax element indicating the PROF for affine prediction is disabled for the current picture, the PROF for affine prediction is not applied to the affine coded block in the current picture; and

responsive to the PROF for affine prediction being determined as applied to the affine coded block:

generating spatial gradients g x (i,j) and g y (i,j) at a sample location (i,j) in the affine coded block in a prediction sample I(i,j);

generating a prediction refinement ΔI(i,j) based on the spatial gradients g x (i,j) and g y (i,j); and

adding the prediction refinement ΔI(i,j) to the prediction sample I(i,j) to generate a refined prediction sample.