Granted Patent B2

US 12,389,037 · App. 18/193,131 · Granted Aug 12, 2025

Inventors: Kai Zhang (San Diego, CA); Li Zhang (San Diego, CA); Hongbin Liu (Beijing, CN); Jizheng Xu (San Diego, CA); Yue Wang (Beijing, CN)

Assignees: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD.; BYTEDANCE INC.

H04N19/61H04N19/11H04N19/124H04N19/132H04N19/159H04N19/176H04N19/186H04N19/70

Abstract

Claims (67)

1. A method of processing video data, comprising:

determining, for a conversion between a current block of a video and a bitstream of the video, that a secondary transform is applicable to the current block, wherein the secondary transform comprises at least one of a forward secondary transform and an inverse secondary transform, wherein the forward secondary transform is performed between a forward primary transform and a quantization, and the inverse secondary transform is performed between a de-quantization and an inverse primary transform;

determining, in response to a dimension of the current block satisfying a first condition, that the secondary transform with an 8×8 secondary transform size is applicable to a first single top-left sub-block of the current block with a dimension of 8×8, and wherein the first condition requires that the dimension of the current block is W1×H1, and wherein H1≥8 and W1≥8;

determining, in response to the dimension of the current block satisfying a second condition, that the secondary transform with a 4×4 secondary transform size is applicable to a second single top-left sub-block of the current block with a dimension of 4×4 and that no secondary transform is applied to a sub-block having a dimension of 4×4 and adjacent to the second single top-left sub-block, and wherein the second condition requires that the dimension of the current block is 4×H1 or W1×4, wherein H1>8 and W1>8; and

performing the conversion based on the determining,

wherein an output value from the inverse secondary transform is constrained within a first range of [min, max] inclusively, wherein min and max are integer values, and wherein min is equal to −(1<<15) and max is equal to (1<<15)−1, and

wherein coefficients after the de-quantization are constrained to a second range of [qmin, qmax] inclusively, wherein qmin is equal to the min of the first range and qmax is equal to the max of the first range, qmin and qmax being integers.

2. The method of claim 1 , wherein a matrix for the secondary transform is selected from four transform sets, and each of the four transform sets consists of two transform matrices.

3. The method of claim 1 , wherein whether to apply the secondary transform depends on a coding mode of a block,

wherein in response to the block being coded with a transform skip mode, the secondary transform is not applied to the block, or

wherein in response to the block being coded with a non intra prediction mode, the secondary transform is not applied to the block.

4. The method of claim 1 , wherein a quantization matrix used in the de-quantization is determined based on whether the inverse secondary transform is applied or not.

5. The method of claim 1 , wherein a clipping operation is applied to clip the output value within the first range of [min, max] inclusively.

6. The method of claim 1 , wherein the inverse secondary transform is applied to de-quantized transformed coefficients of the current block.

7. The method of claim 1 , wherein the secondary transform comprises a low frequency non-separable transform.

8. The method of claim 1 , wherein the conversion includes encoding the video into the bitstream.

9. The method of claim 1 , wherein the conversion includes decoding the video from the bitstream.

10. An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

determine, for a conversion between a current block of a video and a bitstream of the video, that a secondary transform is applicable to the current block, wherein the secondary transform comprises at least one of a forward secondary transform and an inverse secondary transform, wherein the forward secondary transform is performed between a forward primary transform and a quantization, and the inverse secondary transform is performed between a de-quantization and an inverse primary transform;

determine, in response to a dimension of the current block satisfying a first condition, that the secondary transform with an 8×8 secondary transform size is applicable to a first single top-left sub-block of the current block with a dimension of 8×8, and wherein the first condition requires that the dimension of the current block is W1×H1, and wherein H1≥8 and W1≥8;

determine, in response to the dimension of the current block satisfying a second condition, that the secondary transform with a 4×4 secondary transform size is applicable to a second single top-left sub-block of the current block with a dimension of 4×4 and that no secondary transform is applied to a sub-block having a dimension of 4×4 and adjacent to the second single top-left sub-block, and wherein the second condition requires that the dimension of the current block is 4×H1 or W1×4, wherein H1>8 and W1>8; and

perform the conversion based on the determining,

wherein an output value from the inverse secondary transform is constrained within a first range of [min, max] inclusively, wherein min and max are integer values, and wherein min is equal to −(1<<15) and max is equal to (1<<15)−1, and

wherein coefficients after the de-quantization are constrained to a second range of [qmin, qmax]inclusively, wherein qmin is equal to the min of the first range and qmax is equal to the max of the first range, qmin and qmax being integers.

11. The apparatus of claim 10 , wherein a matrix for the secondary transform is selected from four transform sets, and each of the four transform sets consists of two transform matrices,

whether to apply the secondary transform depends on a coding mode of a block,

wherein in response to the block being coded with a transform skip mode, the secondary transform is not applied to the block, or

wherein in response to the block being coded with a non intra prediction mode, the secondary transform is not applied to the block, and

wherein the secondary transform comprises a low frequency non-separable transform.

12. The apparatus of claim 10 , wherein a quantization matrix used in the de-quantization is determined based on whether the inverse secondary transform is applied or not,

wherein a clipping operation is applied to clip the output value within the first range of [min, max] inclusively, and

wherein the inverse secondary transform is applied to de-quantized transformed coefficients of the current block.

13. The apparatus of claim 10 , wherein the conversion includes encoding the video into the bitstream.

14. The apparatus of claim 10 , wherein the conversion includes decoding the video from the bitstream.

15. A non-transitory computer-readable storage medium storing instructions that cause a processor to:

determine, for a conversion between a current block of a video and a bitstream of the video, that a secondary transform is applicable to the current block, wherein the secondary transform comprises at least one of a forward secondary transform and an inverse secondary transform, wherein the forward secondary transform is performed between a forward primary transform and a quantization, and the inverse secondary transform is performed between a de-quantization and an inverse primary transform;

determine, in response to a dimension of the current block satisfying a first condition, that the secondary transform with an 8×8 secondary transform size is applicable to a first single top-left sub-block of the current block with a dimension of 8×8, and wherein the first condition requires that the dimension of the current block is W1×H1, and wherein H1≥8 and W1≥8;

determine, in response to the dimension of the current block satisfying a second condition, that the secondary transform with a 4×4 secondary transform size is applicable to a second single top-left sub-block of the current block with a dimension of 4×4 and that no secondary transform is applied to a sub-block having a dimension of 4×4 and adjacent to the second single top-left sub-block, and wherein the second condition requires that the dimension of the current block is 4×H1 or W1×4, wherein H1>8 and W1>8; and

perform the conversion based on the determining,

wherein an output value from the inverse secondary transform is constrained within a first range of [min, max] inclusively, wherein min and max are integer values, and wherein min is equal to −(1<<15) and max is equal to (1<<15)−1, and

wherein coefficients after the de-quantization are constrained to a second range of [qmin, qmax] inclusively, wherein qmin is equal to the min of the first range and qmax is equal to the max of the first range, qmin and qmax being integers.

16. The non-transitory computer-readable storage medium of claim 15 , wherein a matrix for the secondary transform is selected from four transform sets, and each of the four transform sets consists of two transform matrices,

whether to apply the secondary transform depends on a coding mode of a block,

wherein in response to the block being coded with a transform skip mode, the secondary transform is not applied to the block, or

wherein in response to the block being coded with a non intra prediction mode, the secondary transform is not applied to the block, and

wherein the secondary transform comprises a low frequency non-separable transform.

17. The non-transitory computer-readable storage medium of claim 15 ,

wherein a quantization matrix used in the de-quantization is determined based on whether the inverse secondary transform is applied or not,

wherein a clipping operation is applied to clip the output value within the first range of [min, max] inclusively, and

wherein the inverse secondary transform is applied to de-quantized transformed coefficients of the current block.

18. A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:

determining that a secondary transform is applicable to a current block of the video, wherein the secondary transform comprises at least one of a forward secondary transform and an inverse secondary transform, wherein the forward secondary transform is performed between a forward primary transform and a quantization, and the inverse secondary transform is performed between a de-quantization and an inverse primary transform;

determining, in response to a dimension of the current block satisfying a first condition, that the secondary transform with an 8×8 secondary transform size is applicable to a first single top-left sub-block of the current block with a dimension of 8×8, and wherein the first condition requires that the dimension of the current block is W1×H1, and wherein H1≥8 and W1≥8;

determining, in response to the dimension of the current block satisfying a second condition, that the secondary transform with a 4×4 secondary transform size is applicable to a second single top-left sub-block of the current block with a dimension of 4×4 and that no secondary transform is applied to a sub-block having a dimension of 4×4 and adjacent to the second single top-left sub-block, and wherein the second condition requires that the dimension of the current block is 4×H1 or W1×4, wherein H1>8 and W1>8; and

generating the bitstream of the video based on the determining,

wherein an output value from the inverse secondary transform is constrained within a first range of [min, max] inclusively, wherein min and max are integer values, and wherein min is equal to −(1<<15) and max is equal to (1<<15)−1, and

wherein coefficients after the de-quantization are constrained to a second range of [qmin, qmax] inclusively, wherein qmin is equal to the min of the first range and qmax is equal to the max of the first range, qmin and qmax being integers.

19. The non-transitory computer-readable recording medium of claim 18 ,

wherein a matrix for the secondary transform is selected from four transform sets, and each of the four transform sets consists of two transform matrices,

whether to apply the secondary transform depends on a coding mode of a block,

wherein in response to the block being coded with a transform skip mode, the secondary transform is not applied to the block, or

wherein in response to the block being coded with a non intra prediction mode, the secondary transform is not applied to the block, and

wherein the secondary transform comprises a low frequency non-separable transform.

20. The non-transitory computer-readable recording medium of claim 18 ,

wherein a quantization matrix used in the de-quantization is determined based on whether the inverse secondary transform is applied or not,

wherein a clipping operation is applied to clip the output value within the first range of [min, max] inclusively, and

wherein the inverse secondary transform is applied to de-quantized transformed coefficients of the current block.

Assignments (2)

Priority Claims (1)

Continuity (3)

References Cited (92)