Deblocking of transform sub-block boundaries

1. A method for deblocking at least one boundary of a coding unit, the method comprising:

determining that the coding unit uses a sub-block transform, wherein the sub-block transform generates a transform sub-block boundary within the coding unit, thereby forming at least a transform sub-block within the coding unit, and wherein the sub-block transform comprises Sub-Block Transforms and/or Intra Sub Partitions;

as a result of determining that the coding unit uses the sub-block transform, determining a maximum filter length based on a dimension of the transform sub-block within the coding unit, wherein the dimension of the transform sub-block within the coding unit is a width of the transform sub-block when the boundary is vertical boundary or a height of the transform sub-block when the boundary is a horizontal boundary, wherein the determining comprises comparing the dimension of the transform sub-block with a predetermined size threshold, wherein the maximum filter length indicates a maximum number of samples to modify when deblocking a boundary of the coding unit, and wherein the determined maximum filter length is reduced if the coding unit uses prediction sub-blocks comprising a width smaller than 16 for deblocking of a vertical boundary or height smaller than 16 for deblocking of a horizontal boundary; and

deblocking the boundary of the coding unit based on the determined maximum filter length, wherein the boundary of the coding unit is aligned with a coding unit boundary, a transform sub-block boundary, or a prediction sub-block boundary, wherein

the maximum filter length on the side of the boundary which is inside the coding unit that uses sub-block transform is set to 7 when the transform sub-block boundary is parallel with a coding unit boundary and is at least equal to or larger than a predetermined size threshold in samples from the transform sub-block boundary.

2. The method of claim 1 , wherein the maximum filter length is reduced to 5 when the coding unit uses prediction sub-blocks which boundaries are separated by 8 samples or less.

3. The method of claim 1 , wherein the predetermined size threshold is set to 32.

4. The method of claim 1 , wherein the boundary is aligned with a coding unit boundary or a transform sub-block boundary and deblocking is applied on a 4×4 grid and the max filter length for the side of the boundary that is inside the coding unit is determined to be larger than 3 and the other side of the boundary is inside a coding unit or sub-block transform unit of size 4 the maximum filter length on that side is determined to be less than 3.

5. The method of claim 1 , wherein the maximum transform size is equal to or larger than 32.

6. A computer program product comprising a non-transitory computer readable medium storing a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform the method of claim 1 .

7. An apparatus for deblocking at least one boundary of a coding unit, the apparatus comprising:

memory; and

processing circuitry coupled to the memory, wherein the apparatus is configured to:

determine that the coding unit uses a sub-block transform, wherein the sub-block transform generates a transform sub-block boundary within the coding unit, thereby forming at least a transform sub-block within the coding unit, and wherein the sub-block transform comprises Sub-Block Transforms and/or Intra Sub Partitions;

as a result of determining that the coding unit uses the sub-block transform, determine a maximum filter length based on a dimension of the transform sub-block within the coding unit, wherein the dimension of the transform sub-block within the coding unit is a width of the transform sub-block when the boundary is vertical boundary or a height of the transform sub-block when the boundary is a horizontal boundary, wherein the determining comprises comparing the dimension of the transform sub-block with a predetermined size threshold, wherein the maximum filter length indicates a maximum number of samples to modify when deblocking a boundary of the coding unit, wherein the determined max filter length is reduced if the coding unit uses prediction sub-blocks comprising a width smaller than 16 for deblocking of a vertical boundary or height smaller than 16 for deblocking of a horizontal boundary; and

deblock the boundary of the coding unit based on the determined maximum filter length, wherein the boundary of the coding unit is aligned with a coding unit boundary, a transform sub-block boundary, or a prediction sub-block boundary, wherein

the maximum filter length on the side of the boundary which is inside the coding unit that uses sub-block transform is set to 7 when the transform sub-block boundary is parallel with a coding unit boundary and is at least equal to or larger than a predetermined size threshold in samples from the transform sub-block boundary.

8. The apparatus of claim 7 , wherein the maximum filter length is reduced to 5 when the coding unit uses prediction sub-blocks which boundaries are separated by 8 samples or less.

9. The apparatus of claim 7 , wherein the predetermined size threshold is set to 32.

10. The apparatus of claim 7 , wherein the boundary is aligned with a coding unit boundary or a transform sub-block boundary and deblocking is applied on a 4×4 grid and the max filter length for the side of the boundary that is inside the coding unit is determined to be larger than 3 and the other side of the boundary is inside a coding unit or sub-block transform unit of size 4 the maximum filter length on that side is determined to be less than 3.

11. The apparatus of claim 7 , wherein the maximum transform size is equal to or larger than 32.