Sample determination for adaptive loop filtering

1. A method of processing video data, comprising:

determining, for a conversion between a picture of a video comprising one or more coding tree blocks and a bitstream of the video, that a first filtering process is applied to a current coding tree block of the one or more coding tree blocks;

determining a classification of an M×N region of the current coding tree block based on gradients derived from a P×Q region including the M×N region according to a rule, wherein the rule disallows use of a sample in the P×Q region across one or more boundaries of multiple kinds of video regions, and wherein the picture is divided into multiple kinds of video regions and the one or more boundaries at least comprise a 360-degree virtual boundary or a picture boundary, where M, N, P, Q are positive integers, P is greater than M and Q is greater than N; and

performing the conversion by applying the first filtering process according to the determined classification, wherein the first filtering process comprises:

deriving a filtering coefficient set for the M×N region based on the determined classification;

determining a clipping value set for the M×N region; and

performing a filtering operation based on the filtering coefficient set and the clipping value set.

2. The method of claim 1 , wherein a one-side padding process is applied to generate one or more virtual samples for the sample across the one or more boundaries, wherein the one or more virtual samples are copied from available samples inside a video region where the M×N region is located in.

3. The method of claim 1 , wherein the one or more boundaries conditionally further comprise at least one of: a slice boundary, a tile boundary, or a sub-picture boundary.

4. The method of claim 2 , further comprising determining whether a virtual boundary is enabled for the current coding tree block based on a relationship between a bottom boundary of the current coding tree block and a bottom boundary of the picture.

5. The method of claim 4 , wherein the virtual boundary is enabled in case the bottom boundary of the current coding tree block is not the bottom boundary of the picture.

6. The method of claim 4 , wherein the virtual boundary is used to determine the classification, and wherein samples outside the virtual boundary are not used to determine the classification.

7. The method of claim 1 , wherein the M×N region is a 4*4 region and the P×Q region is a 10*10 region.

8. The method of claim 1 , wherein 1:2 sub sampling is used when determining the classification.

9. The method of claim 1 , wherein the first filtering process is an adaptive loop filtering process.

10. The method of claim 1 , wherein the conversion includes encoding the video into the bitstream.

11. The method of claim 1 , wherein the conversion includes decoding the bitstream into the video.

12. An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

determine, for a conversion between a picture of a video comprising one or more coding tree blocks and a bitstream of the video, that a first filtering process is applied to a current coding tree block of the one or more coding tree blocks;

determine a classification of an M×N region of the current coding tree block based on gradients derived from a P×Q region including the M×N region according to a rule, wherein the rule disallows use of a sample in the P×Q region across one or more boundaries of multiple kinds of video regions, and wherein the picture is divided into multiple kinds of video regions and the one or more boundaries at least comprise a 360-degree virtual boundary or a picture boundary, where M, N, P, Q are positive integers, P is greater than M and Q is greater than N; and

perform the conversion by applying the first filtering process according to the determined classification, wherein the first filtering process comprises:

deriving a filtering coefficient set for the M×N region based on the determined classification;

determining a clipping value set for the M×N region; and

performing a filtering operation based on the filtering coefficient set and the clipping value set.

13. The apparatus of claim 12 , wherein a one-side padding process is applied to generate one or more virtual samples for the sample across the one or more boundaries, wherein the one or more virtual samples are copied from available samples inside a video region where the M×N region is located in; and

wherein the one or more boundaries conditionally further comprise at least one of: a slice boundary, a tile boundary, or a sub-picture boundary.

14. The apparatus of claim 12 , wherein the processor is further caused to determine whether a virtual boundary is enabled for the current coding tree block based on a relationship between a bottom boundary of the current coding tree block and a bottom boundary of the picture;

wherein the virtual boundary is enabled in case the bottom boundary of the current coding tree block is not the bottom boundary of the picture;

wherein the virtual boundary is used to determine the classification, and wherein samples outside the virtual boundary are not used to determine the classification; and

wherein the first filtering process is an adaptive loop filtering process.

15. A non-transitory computer-readable storage medium storing instructions that cause a processor to:

determine, for a conversion between a picture of a video comprising one or more coding tree blocks and a bitstream of the video, that a first filtering process is applied to a current coding tree block of the one or more coding tree blocks;

determine a classification of an M×N region of the current coding tree block based on gradients derived from a P×Q region including the M×N region according to a rule, wherein the rule disallows use of a sample in the P×Q region across one or more boundaries of multiple kinds of video regions, and wherein the picture is divided into multiple kinds of video regions and the one or more boundaries at least comprise a 360-degree virtual boundary or a picture boundary, where M, N, P, Q are positive integers, P is greater than M and Q is greater than N; and

perform the conversion by applying the first filtering process according to the determined classification, wherein the first filtering process comprises:

deriving a filtering coefficient set for the M×N region based on the determined classification;

determining a clipping value set for the M×N region; and

performing a filtering operation based on the filtering coefficient set and the clipping value set.

16. The non-transitory computer-readable storage medium of claim 15 , wherein a one-side padding process is applied to generate one or more virtual samples for the sample across the one or more boundaries, wherein the one or more virtual samples are copied from available samples inside a video region where the M×N region is located in; and

wherein the one or more boundaries conditionally further comprise at least one of: a slice boundary, a tile boundary, or a sub-picture boundary.

17. The non-transitory computer-readable storage medium of claim 15 , wherein the processor is further caused to determine whether a virtual boundary is enabled for the current coding tree block based on a relationship between a bottom boundary of the current coding tree block and a bottom boundary of the picture;

wherein the virtual boundary is enabled in case the bottom boundary of the current coding tree block is not the bottom boundary of the picture;

wherein the virtual boundary is used to determine the classification, and wherein samples outside the virtual boundary are not used to determine the classification; and

wherein the first filtering process is an adaptive loop filtering process.

18. A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:

determining, for a picture of a video comprising one or more coding tree blocks, that a first filtering process is applied to a current coding tree block of the one or more coding tree blocks;

determining a classification of an M×N region of the current coding tree block based on gradients derived from a P×Q region including the M×N region according to a rule, wherein the rule disallows use of a sample in the P×Q region across one or more boundaries of multiple kinds of video regions, and wherein the picture is divided into multiple kinds of video regions and the one or more boundaries at least comprise a 360-degree virtual boundary or a picture boundary, where M, N, P, Q are positive integers, P is greater than M and Q is greater than N;

generating the bitstream by applying the first filtering process according to the determined classification, wherein the first filtering process comprises:

deriving a filtering coefficient set for the M×N region based on the determined classification;

determining a clipping value set for the M×N region; and

performing a filtering operation based on the filtering coefficient set and the clipping value set.

19. The non-transitory computer-readable storage medium of claim 18 , wherein a one-side padding process is applied to generate one or more virtual samples for the sample across the one or more boundaries, wherein the one or more virtual samples are copied from available samples inside a video region where the M×N region is located in; and

wherein the one or more boundaries conditionally further comprise at least one of: a slice boundary, a tile boundary, or a sub-picture boundary.

20. The non-transitory computer-readable storage medium of claim 18 , wherein the method further comprises determining whether a virtual boundary is enabled for the current coding tree block based on a relationship between a bottom boundary of the current coding tree block and a bottom boundary of the picture;

wherein the virtual boundary is enabled in case the bottom boundary of the current coding tree block is not the bottom boundary of the picture;

wherein the virtual boundary is used to determine the classification, and wherein samples outside the virtual boundary are not used to determine the classification; and

wherein the first filtering process is an adaptive loop filtering process.