Methods and devices for intra sub-partition coding mode

1. An intra sub-partition (ISP) method of decoding a video signal, the method comprising:

partitioning a block of video data into a plurality of sub-partitions, wherein each sub-partition comprises N×M samples, where N and M are number of samples in horizontal or vertical direction;

obtaining a number of most probable intra predictions based on intra prediction modes of neighboring blocks of a current block and determining an intra prediction mode to predict the samples of the block sub-partitions;

selecting an optimal transform combination from discrete cosine transform II (DCT-II), discrete sine transform VII (DST-VII), and discrete cosine transform VIII (DCT-VIII) transform kernels for each sub-partition, wherein selecting the optimal transform combination comprises applying a multiple transform selection (MTS) mode to the ISP, wherein the optimal transform combination is indicated by one MTS flag and one MTS index;

obtaining residual coefficients of a first sub-partition and deriving dequantized residual coefficients of the first sub-partition by performing inverse-quantization of the residual coefficients of the first sub-partition;

deriving residual samples of the first sub-partition by performing inverse transforms on dequantized residual coefficients of the first sub-partition in horizontal and vertical directions, wherein the inverse transforms are based on the optimal transform combination of the first sub-partition;

deriving prediction samples of the first sub-partition by performing intra prediction of the first sub-partition based on reconstructed samples adjacent to the first sub-partition;

deriving the reconstructed samples of the first sub-partition by adding the residual samples of the first sub-partition to the prediction samples of the first sub-partition; and

deriving prediction samples of a second sub-partition by performing intra prediction of the second sub-partition based on the reconstructed samples of the first sub-partition,

wherein applying the MTS mode to the ISP comprises:

selecting horizontal and vertical transforms by receiving one MTS flag for the block and one MTS index for each sub-partition, wherein the MTS flag is shared by sub-partitions.

2. The method of claim 1 , further comprising:

selecting transform kernels from DCT-II and DST-VII based on a sub-partition size in response to determining that the MTS flag is zero; and

selecting transform kernels from DST-VII and DCT-VIII that are applied in horizontal and vertical directions for one sub-partition based on the value of the corresponding MTS index in response to determining that the MTS flag is not zero.

3. The method of claim 2 , wherein selecting transform kernels from DCT-II and DST-VII based on the sub-partition size further comprises:

in response to determining that the number of samples in one dimension of the sub-partition is equal or smaller than 16, applying the DST-VII transform in the dimension;

in response to determining that the number of samples in one dimension is larger than 16, applying the DCT-II transform in the dimension.

4. A computing device comprising:

one or more processors;

a non-transitory computer-readable memory storing instructions executable by the one or more processors, wherein the one or more processors are configured to:

partition a block of video data into a plurality of sub-partitions, wherein each sub-partition comprises N×M samples where N and M are number of samples in horizontal or vertical direction;

obtain a number of most probable intra predictions based on intra prediction modes of neighboring blocks of a current block and determining an intra prediction mode to predict the samples of the block sub-partitions;

select an optimal transform combination from discrete cosine transform II (DCT-II), discrete sine transform VII (DST-VII), and discrete cosine transform VIII (DCT-VIII) transform kernels for each sub-partition, wherein selecting the optimal transform combination comprises applying a multiple transform selection (MTS) mode to an intra sub-partition (ISP) mode the ISP, wherein the optimal transform combination is indicated by one MTS flag and one MTS index;

obtain residual coefficients of a first sub-partition and deriving dequantized residual coefficients of the first sub-partition by performing inverse-quantization of the residual coefficients of the first sub-partition;

derive residual samples of the first sub-partition by performing inverse transforms on dequantized residual coefficients of the first sub-partition in horizontal and vertical directions, wherein the inverse transforms are based on the optimal transform combination of the first sub-partition;

derive prediction samples of the first sub-partition by performing intra prediction of the first sub-partition based on reconstructed samples adjacent to the first sub-partition;

derive the reconstructed samples of the first sub-partition by adding the residual samples of the first sub-partition to the prediction samples of the first sub-partition; and

derive prediction samples of a second sub-partition by performing intra prediction of the second sub-partition based on the reconstructed samples of the first sub-partition,

wherein the one or more processors are further configured to:

select horizontal and vertical transforms by receiving one MTS flag for the block and one MTS index for each sub-partition, wherein the MTS flag is shared by sub-partitions.

5. The computing device of claim 4 , wherein the one or more processors are further configured to:

select transform kernels from DCT-II and DST-VII based on a sub-partition size in response to determining that the MTS flag is zero; and

select transform kernels from DST-VII and DCT-VIII that are applied in horizontal and vertical directions for one sub-partition based on the value of the corresponding MTS index in response to determining that the MTS flag is not zero.

6. The computing device of claim 5 , wherein the one or more processors configured to select transform kernels from DCT-II and DST-VII based on the sub-partition size are further configured to:

in response to determining that the number of samples in one dimension of the sub-partition is equal or smaller than 16, apply the DST-VII transform in the dimension;

in response to determining that the number of samples in one dimension is larger than 16, apply the DCT-II transform in the dimension.

7. A non-transitory computer-readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the computing device to perform:

partitioning a block of video data into a plurality of sub-partitions, wherein each sub-partition comprises N×M samples where N and M are number of samples in horizontal or vertical direction;

obtaining a number of most probable intra predictions based on intra prediction modes of neighboring blocks of a current block and determining an intra prediction mode to predict the samples of the block sub-partitions;

selecting an optimal transform combination from discrete cosine transform II (DCT-II), discrete sine transform VII (DST-VII), and discrete cosine transform VIII (DCT-VIII) transform kernels for each sub-partition, wherein selecting the optimal transform combination comprises applying a multiple transform selection (MTS) mode to an intra sub-partition (ISP) mode the ISP, wherein the optimal transform combination is indicated by one MTS flag and one MTS index;

obtaining residual coefficients of a first sub-partition and deriving dequantized residual coefficients of the first sub-partition by performing inverse-quantization of the residual coefficients of the first sub-partition;

deriving residual samples of the first sub-partition by performing inverse transforms on dequantized residual coefficients of the first sub-partition in horizontal and vertical directions, wherein the inverse transforms are based on the optimal transform combination of the first sub-partition;

deriving prediction samples of the first sub-partition by performing intra prediction of the first sub-partition based on reconstructed samples adjacent to the first sub-partition;

deriving the reconstructed samples of the first sub-partition by adding the residual samples of the first sub-partition to the prediction samples of the first sub-partition; and

deriving prediction samples of a second sub-partition by performing intra prediction of the second sub-partition based on the reconstructed samples of the first sub-partition,

wherein the plurality of programs caused the computing device to perform applying the MTS mode to the ISP further cause the computing device to perform:

selecting horizontal and vertical transforms by receiving one MTS flag for the block and one MTS index for each sub-partition, wherein the MTS flag is shared by sub-partitions.

8. The non-transitory computer readable storage medium of claim 7 , wherein the plurality of programs further cause the computing device to perform:

selecting transform kernels from DCT-II and DST-VII based on a sub-partition size in response to determining that the MTS flag is zero; and

selecting transform kernels from DST-VII and DCT-VIII that are applied in horizontal and vertical directions for one sub-partition based on the value of the corresponding MTS index in response to determining that the MTS flag is not zero.

9. The non-transitory computer readable storage medium of claim 8 , wherein the plurality of programs caused the computing device to perform selecting transform kernels from DCT-II and DST-VII based on the sub-partition size further cause the computing device to perform:

in response to determining that the number of samples in one dimension of the sub-partition is equal or smaller than 16, applying the DST-VII transform in the dimension;

in response to determining that the number of samples in one dimension is larger than 16, applying the DCT-II transform in the dimension.