Deblocking filter for video coding

1. A method of processing video data, comprising:

identifying, for applying a deblocking filtering process to a first video block and a second video block of a video during a conversion between the first and second video blocks and a bitstream of the video, at least one boundary at a vertical edge and/or a horizontal edge of the first and second video blocks, wherein the at least one boundary includes a prediction subblock boundary;

determining a boundary strength of the at least one boundary;

deciding whether to turn on or off a filter for the at least one boundary based on the boundary strength;

selecting a strength of the filter in case the filter is turned on; and

performing the conversion based on the selecting,

wherein the boundary strength is determined based on whether an absolute difference between a horizontal or vertical component of block vectors used in a prediction of the first and second video blocks is greater than or equal to a threshold in case that both the first and second video blocks are coded using an intra block copy prediction mode, wherein the threshold is a non-integer, and

wherein the prediction subblock boundary includes a prediction unit boundary introduced by a subblock-based temporal motion vector predictor mode or an affine motion compensation mode, and wherein the deblocking filtering process is applied on an 8x8 grid for the prediction subblock boundary.

2. The method of claim 1 , wherein the conversion includes encoding the first and second video blocks into the bitstream.

3. The method of claim 1 , wherein the conversion includes decoding the first and second video blocks from the bitstream.

4. The method of claim 1 ,

wherein the at least one boundary further includes a transform boundary or a coding unit boundary,

wherein the boundary strength is equal to 2 in case that at least one of the first video block and the second video block crossing the vertical edge or the horizontal edge is coded using a combined inter-intra prediction mode,

wherein in the combined inter-intra prediction mode, a prediction signal of the at least one of the first and second video blocks is generated at least based on an intra prediction signal and an inter prediction signal.

5. The method of claim 4 , wherein the boundary strength is determined based on whether different reference pictures or a different number of motion vectors are used in the prediction of the two video blocks.

6. The method of claim 4 , wherein a stronger deblocking filter is used in the deblocking filtering process when samples at either one side of the at least one boundary belong to a large block, where a sample belonging to the large block is defined as when width is larger than or equal to 32 for the vertical edge, and when height is larger than or equal to 32 for the horizontal edge.

7. The method of claim 6 , wherein the stronger deblocking filter is a bilinear filter.

8. An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

identify, for applying a deblocking filtering process to a first video block and a second video block of a video during a conversion between the first and second video blocks and a bitstream of the video, at least one boundary at a vertical edge and/or a horizontal edge of the first and second video blocks, wherein the at least one boundary includes a prediction subblock boundary;

determine a boundary strength of the at least one boundary;

decide whether to turn on or off a filter for the at least one boundary based on the boundary strength;

select a strength of the filter in case the filter is turned on; and

perform the conversion based on the selecting, wherein the boundary strength is determined based on whether an absolute difference between a horizontal or vertical component of block vectors used in a prediction of the first and second video blocks is greater than or equal to a threshold in case that both the first and second video blocks are coded using an intra block copy prediction mode, wherein the threshold is a non-integer, and wherein the prediction subblock boundary includes a prediction unit boundary introduced by a subblock-based temporal motion vector predictor mode or an affine motion compensation mode, and wherein the deblocking filtering process is applied on an 8x8 grid for the prediction subblock boundary.

9. The apparatus of claim 8 , wherein the at least one boundary further includes a transform boundary or a coding unit boundary,

wherein the boundary strength is equal to 2 in case that at least one of the first video block and the second video block crossing the vertical edge or the horizontal edge is coded using a combined inter-intra prediction mode,

wherein in the combined inter-intra prediction mode, a prediction signal of the at least one of the first and second video blocks is generated at least based on an intra prediction signal and an inter prediction signal.

10. The apparatus of claim 9 , wherein the boundary strength is determined based on whether different reference pictures or a different number of motion vectors are used in the prediction of the first and second video blocks.

11. The apparatus of claim 9 , wherein a stronger deblocking filter is used in the deblocking filtering process when samples at either one side of the at least one boundary belong to a large block, where a sample belonging to the large block is defined as when width is larger than or equal to 32 for the vertical edge, and when height is larger than or equal to 32 for the horizontal edge.

12. The apparatus of claim 11 , wherein the stronger deblocking filter is a bilinear filter.

13. A non-transitory computer-readable storage medium storing instructions that cause a processor to:

identify, for applying a deblocking filtering process to a first video block and a second video block of a video during a conversion between the first and second video blocks and a bitstream of the video, at least one boundary at a vertical edge and/or a horizontal edge of the first and second video blocks, wherein the at least one boundary includes a prediction subblock boundary;

determine a boundary strength of the at least one boundary;

decide whether to turn on or off a filter for the at least one boundary based on the boundary strength;

select a strength of the filter in case the filter is turned on; and

perform the conversion based on the selecting, wherein the boundary strength is determined based on whether an absolute difference between a horizontal or vertical component of block vectors used in a prediction of the first and second video blocks is greater than or equal to a threshold in case that both the first and second video blocks are coded using an intra block copy prediction mode, wherein the threshold is a non-integer, and wherein the prediction subblock boundary includes a prediction unit boundary introduced by a subblock-based temporal motion vector predictor mode or an affine motion compensation mode, and wherein the deblocking filtering process is applied on an 8 x 8 grid for the prediction subblock boundary.

14. The non-transitory computer-readable storage medium of claim 13 ,

wherein the at least one boundary further includes a transform boundary or a coding unit boundary,

wherein the boundary strength is equal to 2 in case that at least one of the first video block and the second video block crossing the vertical edge or the horizontal edge is coded using a combined inter-intra prediction mode,

wherein in the combined inter-intra prediction mode, a prediction signal of the at least one of the first and second video blocks is generated at least based on an intra prediction signal and an inter prediction signal.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the boundary strength is determined based on whether different reference pictures or a different number of motion vectors are used in the prediction of the first and second video blocks.

16. The non-transitory computer-readable storage medium of claim 14 , wherein a stronger deblocking filter is used in the deblocking filtering process when samples at either one side of the at least one boundary belong to a large block, where a sample belonging to the large block is defined as when width is larger than or equal to 32 for the vertical edge, and when height is larger than or equal to 32 for the horizontal edge.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the stronger deblocking filter is a bilinear filter.

18. The non-transitory computer-readable recording medium of claim 15 , wherein the boundary strength is determined based on whether different reference pictures or a different number of motion vectors are used in a prediction of the two video blocks.

19. The method of claim 18 , wherein the at least one boundary further includes a transform boundary or a coding unit boundary,

wherein the boundary strength is equal to 2 in case that at least one of the first video block and the second video block crossing the vertical edge or the horizontal edge is coded using a combined inter-intra prediction mode,

wherein in the combined inter-intra prediction mode, a prediction signal of the at least one of the first and second video blocks is generated at least based on an intra prediction signal and an inter prediction signal.

20. The method of claim 19 , wherein the boundary strength is determined based on whether different reference pictures or a different number of motion vectors are used in the prediction of the first and second video blocks.