Method and apparatus of transform type assignment for intra sub-partition in video coding

1. A method of prediction for video coding, the method comprising:

receiving input data related to a current block in a current picture at a video encoder side or a video bitstream corresponding to compressed data including the current block in the current picture at a video decoder side;

if Intra Sub-Partition (ISP) mode is used for the current block:

partitioning the current block into multiple sub-blocks horizontally or vertically;

determining a target horizontal transform and a target vertical transform from a candidate transform set for each of the multiple sub-blocks according to a setting group comprising a unified setting, a block setting, or both, wherein the unified setting is applicable to a particular coding mode, and a normal Intra mode, and the block setting corresponds to block width, block height, block ratio, block area or any combination thereof of the sub-block; and

applying the target horizontal transform and the target vertical transform to each of the multiple sub-blocks,

wherein the ISP mode is disabled according to at least one threshold, the disabling being related to block width, block height, or block area of the current block.

2. The method of claim 1 , wherein the target horizontal transform and the target vertical transform belong to a subset comprising any combination of DCT-II, DST-VII, DCT-VIII, and flipped DST-VII.

3. The method of claim 2 , wherein a pair representing (the target horizontal transform, the target vertical transform) corresponds to (DCT-II, DCT-II), (DST-VII, DCT-II), (DCT-II, DST-VII), or (DST-VII, DST-VII).

4. The method of claim 1 , wherein the target horizontal transform and the target vertical transform are determined based on the block width, the block height, the block ratio, the block area or any combination thereof of the current block.

5. The method of claim 4 , wherein the target horizontal transform is changed from DCT-II to another transform type when the block width is smaller than or equal to the block height.

6. The method of claim 4 , wherein the target horizontal transform is changed from DCT-II to DST-VII when the block width is within [ 4 , 16 ], the block width is smaller than or equal to the block height, or both the block width is within [ 4 , 16 ] and the block width is smaller than or equal to the block height.

7. The method of claim 4 , wherein the target vertical transform is changed from DCT-II to DST-VII when the block height is within [ 4 , 16 ], the block height is smaller than or equal to the block width, or both the block height is within [ 4 , 16 ] and the block height is smaller than or equal to the block width.

8. The method of claim 1 , wherein the candidate transform set is the same as a transform set for the normal Intra mode.

9. The method of claim 8 , wherein a shortest codeword for candidate transforms in the candidate transform set is set to DST-VII for both the target horizontal transform and the target vertical transform.

10. The method of claim 1 , wherein the ISP mode uses a same Intra prediction mode list as that of the normal Intra mode.

11. The method of claim 10 , wherein the ISP mode uses a subset of an Intra prediction mode list of the normal Intra mode.

12. The method of claim 1 , wherein the unified setting or the block setting is determined implicitly.

13. The method of claim 1 , wherein a decision for the unified setting or the block setting is signaled at CU (coding unit), CTU (coding tree unit), slice, tile, tile group, SPS (sequence parameter set), or PPS (picture parameter set) level.

14. The method of claim 1 , wherein the setting group further comprises mode setting and the mode setting corresponds to a selected Intra mode for the current block.

15. The method of claim 14 , wherein the selected Intra mode corresponds to a directional mode satisfying (directional modes % 2==1).

16. The method of claim 14 , wherein the selected Intra mode corresponds to a directional mode satisfying (directional modes % 2==0).

17. An apparatus of video coding, the apparatus comprising one or more electronic circuits or processors arranged to:

receive input data related to a current block in a current picture at a video encoder side or a video bitstream corresponding to compressed data including the current block in the current picture at a video decoder side;

if Intra Sub-Partition (ISP) mode is used for the current block:

partition the current block into multiple sub-blocks horizontally or vertically;

determine a target horizontal transform and a target vertical transform from a candidate transform set for each of the multiple sub-blocks according to a setting group comprising a unified setting, a block setting, or both, wherein the unified setting is applicable to a particular coding mode, and a normal Intra mode, and the block setting corresponds to block width, block height, block ratio, block area or any combination thereof of the sub-block; and

apply the target horizontal transform and the target vertical transform to each of the multiple sub-blocks,

wherein the ISP mode is disabled according to at least one threshold, the disabling being related to block width, block height, or block area of the current block.

18. A method of prediction for video coding, the method comprising:

receiving input data related to a current block in a current picture at a video encoder side or a video bitstream corresponding to compressed data including the current block in the current picture at a video decoder side;

determining a target horizontal transform and a target vertical transform from a candidate transform set without a transform index for the current block according to unified setting, wherein the unified setting comprises two or more Intra modes or Intra-related modes; and

applying the target horizontal transform and the target vertical transform to the current block,

wherein the ISP mode is disabled according to at least one threshold, the disabling being related to block width, block height, or block area of the current block.

19. An apparatus of video coding, the apparatus comprising one or more electronic circuits or processors arranged to:

receive input data related to a current block in a current picture at a video encoder side or a video bitstream corresponding to compressed data including the current block in the current picture at a video decoder side;

determine a target horizontal transform and a target vertical transform from a candidate transform set without a transform index for the current block according to unified setting, wherein the unified setting comprises two or more Intra modes or Intra-related modes; and

apply the target horizontal transform and the target vertical transform to the current block,

wherein the ISP mode is disabled according to at least one threshold, the disabling being related to block width, block height, or block area of the current block.

20. The method of claim 1 , wherein when the block width or the block height of the current block is larger than a maximum threshold, or the block size is smaller than a minimum threshold, the ISP mode is disabled.

21. The method of claim 20 , wherein the maximum threshold is a maximum transform block size.

22. The method of claim 20 , wherein the minimum threshold is a square of minimum transform block size.