Determination of set of candidate transforms for video encoding

1. A method for encoding a picture into a data stream using block-based prediction and block-based residual coding, the method comprising:

determining a set of transform candidates for a current residual block out of a set of transforms based on a height and a width of the current residual block and positions of significant transform coefficients, the determined set including a primary transform, the set of transforms including a plurality of multi-stage transforms and the primary transform, each of the multi-stage transforms including the primary transform and a respective non-separable secondary transform applied to primary transform coefficients resulting in respective secondary transform coefficients;

wherein determining the set of transform candidates includes:

determining whether at least one transform coefficient outside a sub-group of transform coefficient positions is significant;

in response to a determination that no transform coefficient outside the sub-group is significant, including the plurality of multi-stage transforms in the determined set; and

in response to a determination that at least one transform coefficient outside the sub-group is significant, omitting the plurality of multi-stage transforms from the determined set;

selecting a transform out of the determined set of transform candidates;

encoding a prediction residual for the current residual block using the selected transform;

selectively signaling the selected transform in the data stream,

wherein selectively signaling the selected transform includes:

when the determined set of transform candidates includes the plurality of multi-stage transforms, signaling an index in the data stream; and

when the plurality of multi-stage transforms is excluded from the determined set of transform candidates, not signaling the index in the data stream.

2. A method of decoding a picture from a data stream using block-based prediction and block-based residual decoding, the method comprising:

determining a set of transform candidates for a current residual block out of a set of transforms based on a height and a width of the current residual block and positions of significant transform coefficients, the determined set including a reverse transform of a primary transform, the set of transforms including reverse transforms of a plurality of multi-stage transforms and the reverse transform of the primary transform, each of the multi-stage transforms including the primary transform and a respective non-separable secondary transform applied to primary transform coefficients resulting in respective secondary transform coefficients, wherein determining the set of transform candidates includes:

determining whether at least one transform coefficient outside a sub-group of transform coefficient positions is significant;

in response to a determination that no transform coefficient outside the sub-group is significant, including the reverse transforms for the plurality of multi-stage transforms in the determined set; and

in response to a determination that at least one transform coefficient outside the sub-group is significant, omitting the reverse transforms for the plurality of multi-stage transforms from the determined set;

selecting a transform from the determined set of transform candidates and decoding a prediction residual within the current residual block, wherein selecting a transform includes selectively decoding the transform by:

when the determined set of transform candidates includes the reverse transforms for the plurality of multi-stage transforms, decoding an index from the data stream; and

when the reverse transforms for the plurality of multi-stage transforms are excluded from the determined set of transform candidates, not decoding the index from the data stream.

3. The method of claim 1 , wherein the primary transform coefficients to which the non-separable secondary transform is applied represent a low-frequency portion of the primary transform.

4. The method of claim 1 , further comprising:

encoding an indication of a last significant transform coefficient position in a predetermined scan order of sequentially traversing transform coefficient positions and

determining the set of transform candidates based on the indication.

5. The method of claim 1 , wherein the primary transform coefficients to which the non-separable secondary transform is applied belong to three 4×4 left/top most coefficient sub-groups.

6. The method of claim 1 , wherein determining the set of transform candidates is further based on an intra prediction mode aggregation.

7. The method of claim 2 , wherein the primary transform coefficients to which the non-separable secondary transform is applied represent a low-frequency portion of the primary transform.

8. The method of claim 2 , further comprising:

decoding an indication of a last significant transform coefficient position in a predetermined scan order of sequentially traversing transform coefficient positions; and

determining the set of transform candidates based on the indication.

9. The method of claim 2 , wherein the sub-group of primary transform coefficients to which the non-separable secondary transform is applied belong to three 4×4 left/top most coefficient sub-groups.

10. The method of claim 2 , wherein determining the set of transform candidates is further based on an intra prediction mode aggregation.

11. The method of claim 2 , wherein it is determined that no transform coefficient outside the sub-group is significant and between 2 and 6 reverse transforms of respective multi-stage transforms are included in the determined set.

12. A video encoder for encoding a picture into a data stream using block-based prediction and block-based residual coding, the video encoder comprising:

at least one processor configured to:

determine a set of transform candidates for a current residual block out of a set of transforms based on a height and a width of the current residual block and positions of significant transform coefficients, the determined set including a primary transform, the set of transforms including a plurality of multi-stage transforms and the primary transform, each of the multi-stage transforms including the primary transform and a respective non-separable secondary transform applied to primary transform coefficients resulting in respective secondary transform coefficients,

wherein the at least one processor is configured to determine the set of transform candidates by:

determining whether at least one transform coefficient outside a sub-group of transform coefficient positions is significant;

in response to a determination that no transform coefficient outside the sub-group is significant, including the plurality of multi-stage transforms in the determined set; and

in response to a determination that at least one transform coefficient outside the sub-group is significant, omitting the plurality of multi-stage transforms from the determined set;

select a transform out of the determined set of transform candidates;

encode a prediction residual for the current residual block using the selected transform; and

selectively signal the selected transform in the data stream,

wherein the at least one processor is configured the selectively signal the selected transform in the data stream by:

when the determined set of transform candidates includes the plurality of multi-stage transforms, signaling an index in the data stream, and

when the plurality of multi-stage transforms is excluded from the determined set of transform candidates, not signaling the index in the data stream.

13. The video encoder of claim 12 , wherein the primary transform coefficients to which the non-separable transform is applied represent a low-frequency portion of the primary transform.

14. The video encoder of claim 12 , wherein the at least one processor is configured to:

encode an indication of a last significant transform coefficient position in a predetermined scan order of sequentially traversing transform coefficient positions; and

determine the set of transform candidates based on the indication.

15. The video encoder of claim 12 , wherein the primary transform coefficients to which the non-separable secondary transform is applied belong to three 4×4 left/top most coefficient sub-groups.

16. The video encoder of claim 12 , wherein the at least one processor is further configured to determine the set of transform candidates further based on an intra prediction mode aggregation.

17. A video decoder for decoding a picture into a data stream using block-based prediction and block-based residual coding, the video decoder comprising:

at least one processor configuredto:

determine a set of transform candidates for a current residual block out of a set of transforms based on a height and a width of the current residual block and positions of significant transform coefficients, the determined set including a reverse transform of a primary transform, the set of transforms including reverse transforms of a plurality of multi-stage transforms and the reverse transform of the primary transform, each of the multi-stage transforms including the primary transform and a respective non-separable secondary transform applied to primary transform coefficients resulting in respective secondary transform coefficients,

wherein the at least one processor is configured to determine the set of transform candidates by:

determining whether at least one transform coefficient outside a sub-group of transform coefficient positions is significant;

in response to a determination that no transform coefficient outside the sub-group is significant, including the reverse transforms for the plurality of multi-stage transforms in the determined set; and

in response to a determination that at least one transform coefficient outside the sub-group is significant, omitting the reverse transforms for the plurality of multi-stage transforms from the determined set;

select a transform from the determined set of transform candidates and decode a prediction residual within the current residual block, wherein the at least one processor is configured to select a transform by selectively decoding the transform by:

when the determined set of transform candidates includes the reverse transforms for the plurality of multi-stage transforms, decoding an index from the data stream, and

when the reverse transforms for the plurality of multi-stage transforms are excluded from the determined set of transform candidates, not decoding the index from the data stream.

18. The video decoder of claim 17 , wherein the primary transform coefficients to which the non-separable secondary transform is applied represent a low-frequency portion of the primary transform.

19. The video decoder of claim 17 , wherein the at least one processor is configured to:

decode an indication of a last significant transform coefficient position in a predetermined scan order of sequentially traversing transform coefficient positions; and

determine the set of transform candidates based on the indication.

20. The video decoder of claim 17 , wherein the primary transform coefficients to which the non-separable secondary transform is applied belong to three 4×4 left/top most coefficient sub-groups.

21. The video decoder of claim 17 , wherein the at least one processor is configured to determine the set of transform candidates further based on an intra prediction mode aggregation.

22. The video decoder of claim 17 , wherein the processor is configured to include between 2 and 6 reverse transforms of respective multistage transforms in the determined set when it is determined that no transform coefficient outside the sub-group is significant.

23. A non-transitory computer readable storage medium having a computer program stored thereon, the computer program, when executed on a processor, causing the processor to perform:

determining a set of transform candidates for a current residual block out of a set of transforms based on a height and a width of the current residual block and positions of significant transform coefficients, the determined set including a primary transform, the set of transforms including a plurality of multi-stage transforms and the primary transform, each of the multi-stage transforms comprising the primary transform and a respective non-separable secondary transform applied to primary transform coefficients resulting in respective secondary transform coefficients,

wherein determining the set of transform candidates includes:

determining whether at least one transform coefficient outside a sub-group of transform coefficient positions is significant;

in response to a determination that no transform coefficient outside the sub-group is significant, including the plurality of multi-stage transforms in the determined set; and

in response to a determination that at least one transform coefficient outside the sub-group is significant, omitting the plurality of multi-stage transforms from the determined set;

selecting a transform out of the determined set of transform candidates;

encoding a prediction residual for the current residual block using the selected transform; and

selectively signaling the selected transform in a data stream,

wherein selectively signaling the selected transform includes:

when the determined set of transform candidates includes the plurality of multi-stage transforms, signaling an index in the data stream, and

when the plurality of multi-stage transforms is excluded from the determined set of transform candidates, not signaling the index in the data stream.

24. The non-transitory computer readable storage medium of claim 23 , wherein the primary transform coefficients to which the non-separable secondary transform is applied represent a low-frequency portion of the primary transform.

25. The non-transitory computer readable storage medium of claim 23 , wherein the computer program, when executed on the processor, causes the processor to further perform:

encoding an indication of a last significant transform coefficient position in a predetermined scan order of sequentially traversing transform coefficient positions; and

determining the set of transform candidates further based on the indication.

26. The non-transitory computer readable storage medium of claim 23 , wherein the primary transform coefficients to which the non-separable secondary transform is applied belong to three 4×4 left/top most coefficient sub-groups.

27. The non-transitory computer readable storage medium of claim 23 , wherein determining the set of transform candidates is further based on an intra prediction mode aggregation.

28. A non-transitory computer readable storage medium having a computer program stored thereon, the computer program, when executed on a processor, causing the processor to perform:

determining a set of transform candidates for a current residual block out of a set of transforms based on a height and a width of the current residual block and positions of significant transform coefficients, the determined set including a reverse transform of a primary transform, the set of transforms including reverse transforms of a plurality of multi-stage transforms and the reverse transform of the primary transform, each of the multi-stage transforms comprising the primary transform and a respective non-separable secondary transform applied to primary transform coefficients resulting in respective secondary transform coefficients,

wherein determining the set of transform candidates includes:

determining whether at least one transform coefficient outside a sub-group of transform coefficient positions is significant;

in response to a determination that no transform coefficient outside the sub-group is significant, including the reverse transforms for the plurality of multi-stage transforms in the determined set; and

in response to a determination that at least one transform coefficient outside the sub-group is significant, omitting the reverse transforms plurality of multi-stage transforms in the determined set;

selecting a transform from the determined set of transform candidates and decoding a prediction residual within the current residual block, wherein selecting a transform includes selectively decoding the transform by:

when the determined set of transform candidates includes the reverse transforms for the plurality of multi-stage transforms, decoding an index from a data stream, and

when the reverse transforms for the plurality of multi-stage transforms are excluded from the determined set of transform candidates, not decoding the index from the data stream.

29. The non-transitory computer readable storage medium of claim 28 , wherein the primary transform coefficients to which the non-separable secondary transform is applied represent a low-frequency portion of the primary transform.

30. The non-transitory computer readable storage medium of claim 28 ,

wherein computer program, when executed on the processor, causes the processor to further perform:

decoding an indication of a last significant transform coefficient position in a predetermined scan order of sequentially traversing transform coefficient positions; and

determining the set of transform candidates based on the indication.

31. The non-transitory computer readable storage medium of claim 28 , wherein the primary transform coefficients to which the non-separable secondary transform is applied belong to three 4×4 left/top most coefficient sub-groups.

32. The non-transitory computer readable storage medium of claim 28 , wherein determining the set of transform candidates is further based on an intra prediction mode aggregation.

33. The non-transitory computer readable storage medium of claim 28 , wherein it is determined that no significant coefficient outside the sub-group is significant and the reverse transforms of between 2 and 6 respective multi-stage transforms are included in the determined set.