Method and apparatus for improved compound orthonormal transform

1. A method of controlling residual coding for decoding or encoding of a video sequence, the method being performed by at least one processor, and the method comprising:

determining whether a transform size of 8-point or smaller of a primary transform is to be used for the residual coding of a coded block of the video sequence;

in response to the transform size of 8-point or smaller of the primary transform being determined to be used, automatically performing the residual coding of the coded block, using, as the primary transform, a first transform set consisting both discrete sine transform (DST)-4 and discrete cosine transform (DCT)-4;

in response to a transform size of 16-point or larger of the primary transform being determined to be used, automatically performing the residual coding of the coded block, using, as the primary transform, a second transform set consisting both DST-7 and DCT-8;

in response to the residual coding of the coded block being automatically performed, automatically determining whether a secondary transform is to be used for the residual coding of the coded block of which the residual coding is automatically performed using the primary transform;

in response to the secondary transform being determined to be used, automatically determining whether the transform size of 8-point or smaller of the secondary transform is to be used for the residual coding of the coded block;

in response to the transform size of 8-point or smaller of the secondary transform being determined to be used, automatically performing the residual coding of the coded block of which the residual coding is automatically performed using the primary transform, using, as the secondary transform, the first transform set consisting both the DST-4 and the DCT-4; and

in response to the transform size of 16-point or larger of the secondary transform being determined to be used, automatically performing the residual coding of the coded block of which the residual coding is automatically performed using the primary transform, using, as the secondary transform, the second transform set consisting both the DST-7 and the DCT-8.

2. The method of claim 1 , wherein the primary transform is an adaptive multiple transform, and

the secondary transform is a non-separable secondary transform.

3. The method of claim 1 , wherein the performing the residual coding comprises coding residual blocks of the coded block that is intra or inter coded.

4. An apparatus for controlling residual coding for decoding or encoding of a video sequence, the apparatus comprising:

at least one memory configured to store computer program code; and

at least one processor configured to access the at least one memory and operate according to the computer program code, the computer program code comprising:

first determining code configured to cause the at least one processor to determine whether a transform size of 8-point or smaller of a primary transform is to be used for the residual coding of a coded block of the video sequence;

performing code configured to cause the at least one processor to:

in response to the transform size of 8-point or smaller of the primary transform being determined to be used, automatically perform the residual coding of the coded block, using, as the primary transform, a first transform set consisting both discrete sine transform (DST)-4 and discrete cosine transform (DCT)-4; and

in response to a transform size of 16-point or larger of the primary transform being determined to be used, automatically perform the residual coding of the coded block, using, as the primary transform, a second transform set consisting both DST-7 and DCT-8; and

second determining code configured to cause the at least one processor to, in response to the residual coding of the coded block being automatically performed, automatically determine whether a secondary transform is to be used for the residual coding of the coded block of which the residual coding is performed using the primary transform,

wherein the first determining code is further configured to cause the at least one processor to, in response to the secondary transform being determined to be used, automatically determine whether the transform size of 8-point or smaller of the secondary transform is to be used for the residual coding of the coded block, and

wherein the performing code is further configured to cause the at least one processor to:

in response to the transform size of 8-point or smaller of the secondary transform being determined to be used, automatically perform the residual coding of the coded block of which the residual coding is automatically performed using the primary transform, using, as the secondary transform, the first transform set consisting both the DST-4 and the DCT-4, and

in response to the transform size of 16-point or larger of the secondary transform being determined to be used, automatically perform the residual coding of the coded block of which the residual coding is automatically performed using the primary transform, using, as the secondary transform, the second transform set consisting both the DST-7 and the DCT-8.

5. The apparatus of claim 4 , wherein the primary transform is an adaptive multiple transform, and

the secondary transform is a non-separable secondary transform.

6. The apparatus of claim 4 , wherein the performing code is further configured to cause the at least one processor to code residual blocks of the coded block that is intra or inter coded.

7. A non-transitory computer-readable storage medium storing instructions that cause at least one processor to:

determine whether a transform size of 8-point or smaller of a primary transform is to be used for residual coding of a coded block of a video sequence;

in response to the transform size of 8-point or smaller of the primary transform being determined to be used, automatically perform the residual coding of the coded block, using, as the primary transform, a first transform set consisting both discrete sine transform (DST)-4 and discrete cosine transform (DCT)-4;

in response to a transform size of 16-point or larger of the primary transform being determined to be used, automatically perform the residual coding of the coded block, using, as the primary transform, a second transform set consisting both DST-7 and DCT-8;

in response to the residual coding of the coded block being automatically performed, automatically determine whether a secondary transform is to be used for the residual coding of the coded block of which the residual coding is automatically performed using the primary transform;

in response to the secondary transform being determined to be used, automatically determine whether the transform size of 8-point or smaller of the secondary transform is to be used for the residual coding of the coded block;

in response to the transform size of 8-point or smaller of the secondary transform being determined to be used, automatically perform the residual coding of the coded block of which the residual coding is automatically performed using the primary transform, using, as the secondary transform, the first transform set consisting both the DST-4 and the DCT-4; and

in response to the transform size of 16-point or larger of the secondary transform being determined to be used, automatically perform the residual coding of the coded block of which the residual coding is automatically performed using the primary transform, using, as the secondary transform, the second transform set consisting both the DST-7 and the DCT-8.

8. The non-transitory computer-readable storage medium of claim 7 , wherein the instructions further cause the at least one processor to code residual blocks of the coded block that is intra or inter coded.