Extended low-frequency non-separable transform (LFNST) designs with worst-case complexity handling

1. A method of decoding video data, the method comprising:

determining a number of allowed non-zero coefficients for a block of video data based on a size of the block;

obtaining a set of dequantized coefficients for the block of video data, wherein the set of dequantized coefficients comprises a first subset of dequantized coefficients that includes non-zero dequantized coefficients and a second subset of dequantized coefficients that includes all zero coefficients

determining an intra prediction mode for the block of video data;

based on the intra prediction mode, determining a set of inverse LFNST candidates from a plurality of sets, wherein the plurality of sets includes 35 sets, and the determining comprising determining an index indicative of the set of inverse LFNST candidates from the plurality of sets based on the inter prediction mode, and wherein determining the index based on the intra prediction mode comprises:

for values of the intra prediction mode between 0 and 34, determining the index to be equal to the value of the intra prediction mode; and

for values of the intra prediction mode between 35 and 66, determining the index to be equal to 68 minus the value of the intra prediction mode;

selecting an inverse LFNST from the determined set of inverse LFNST candidates,

applying the inverse low-frequency non-separable transform (LFNST) to the first subset of dequantized coefficients to determine a first intermediate subset of coefficients; and

applying an inverse separable transform to the first intermediate subset of coefficients and at least a portion of the second subset of coefficients to determine a block of reconstructed residual values.

2. The method of claim 1 , wherein the set of inverse LFNST candidates includes 3 candidates.

3. The method of claim 1 , wherein the first subset of dequantized coefficients comprises M dequantized coefficients, and wherein first intermediate subset of coefficients comprises N coefficients, wherein M and N are integer values.

4. The method of claim 2 , wherein M=16 and N=64.

5. The method of claim 2 , wherein M=64 and N=64.

6. The method of claim 2 , further comprising:

determining the number of allowed non-zero coefficients according to the equation NZ=floor((WCM×nTbW×nTbH)/(N×N)), wherein nTbH represents a height of the block of reconstructed values, nTbW represents a width of the block of reconstructed values, NZ equals the number of allowed non-zero coefficients and WCM represents a constant value.

7. The method of claim 6 , wherein the first subset of dequantized coefficients has NZ coefficients.

8. The method of claim 6 , wherein M=2×NZ and N=64.

9. The method of claim 1 , further comprising:

adding the block of reconstructed residual values to a prediction block to form a reconstructed block;

applying one or more filters to the reconstructed block to determine a filtered reconstructed block; and

outputting decoded video data that includes the filtered reconstructed block.

10. The method of claim 1 , wherein the method is performed as part of a video encoding process.

11. A device for decoding video data, the device comprising:

a memory configured to store video data;

one or more processors implemented in circuitry and configured to:

determine a number of allowed non-zero coefficients for a block of video data based on a size of the block;

obtain a set of dequantized coefficients for the block of video data, wherein the set of dequantized coefficients comprises a first subset of dequantized coefficients that includes non-zero dequantized coefficients and a second subset of dequantized coefficients that includes all zero coefficients;

determine an intra prediction mode for the block of video data;

based on the intra prediction mode, determine a set of inverse LFNST candidates from a plurality of sets, wherein the plurality of sets includes 35 sets, wherein to determine the set, the one or more processors is configured to determine an index indicative of the set of inverse LFNST candidates from the plurality of sets based on the inter prediction mode, and wherein to

determine the index based on the intra prediction mode, the one or more processors is configured to:

for values of the intra prediction mode between 0 and 34, determine the index to be equal to the value of the intra prediction mode; and

for values of the intra prediction mode between 35 and 66, determine the index to be equal to 68 minus the value of the intra prediction mode;

select an inverse LFNST from the determined set of inverse LFNST candidates;

apply the inverse low-frequency non-separable transform (LFNST) to the first subset of dequantized coefficients to determine a first intermediate subset of coefficients; and

apply an inverse separable transform to the first intermediate subset of coefficients and at least a portion of the second subset of coefficients to determine a block of reconstructed residual values.

12. The device of claim 11 , wherein the set of inverse LFNST candidates includes 3 candidates.

13. The device of claim 11 , wherein the first subset of dequantized coefficients comprises M dequantized coefficients, and wherein first intermediate subset of coefficients comprises N coefficients, wherein M and N are integer values.

14. The device of claim 13 , wherein M=16 and N=64.

15. The device of claim 13 , wherein M=64 and N=64.

16. The device of claim 13 , wherein the one or more processors are further configured to:

determine the number of allowed non-zero coefficients according to the equation NZ=floor((WCM×nTbW×nTbH)/(N×N)), wherein nTbH represents a height of the block of reconstructed values, nTbW represents a width of the block of reconstructed values, NZ equals the number of allowed non-zero coefficients and WCM represents a constant value.

17. The device of claim 13 , wherein the first subset of dequantized coefficients has NZ coefficients.

18. The device of claim 13 , wherein M=2×NZ and N=64.

19. The device of claim 11 , wherein the one or more processors are further configured to:

add the block of reconstructed residual values to a prediction block to form a reconstructed block;

apply one or more filters to the reconstructed block to determine a filtered reconstructed block; and

output decoded video data that includes the filtered reconstructed block.

20. The device of claim 11 , wherein the device comprises a wireless communication device, further comprising a receiver configured to receive encoded video data.

21. The device of claim 20 , wherein the wireless communication device comprises a telephone handset and wherein the receiver is configured to demodulate, according to a wireless communication standard, a signal comprising the encoded video data.

22. The device of claim 11 , further comprising:

a display configured to display decoded video data.

23. The device of claim 11 , wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

24. The device of claim 11 , further comprising:

a camera configured to capture video data.

25. The device of claim 11 , wherein the device comprises a wireless communication device, further comprising a transmitter configured to transmit encoded video data.

26. The device of claim 25 , wherein the wireless communication device comprises a telephone handset and wherein the transmitter is configured to modulate, according to a wireless communication standard, a signal comprising the encoded video data.

27. A non-transitory computer-readable storage medium storing instructions that when executed by one or more processors cause the one or more processors to:

determine a number of allowed non-zero coefficients for a block of video data based on a size of the block;

obtain a set of dequantized coefficients for the block of video data, wherein the set of dequantized coefficients comprises a first subset of dequantized coefficients that includes non-zero dequantized coefficients and a second subset of dequantized coefficients that includes all zero coefficients;

determine an intra prediction mode for the block of video data;

based on the intra prediction mode, determine a set of inverse LFNST candidates from a plurality of sets, wherein the plurality of sets includes 35 sets, wherein to determine the set, the instructions cause the one or more processors to determine an index indicative of the set of inverse LFNST candidates from the plurality of sets based on the inter prediction mode, and wherein to determine the index based on the intra prediction mode, the instructions cause the one or more processors to:

for values of the intra prediction mode between 0 and 34, determine the index to be equal to the value of the intra prediction mode; and

for values of the intra prediction mode between 35 and 66, determine the index to be equal to 68 minus the value of the intra prediction mode;

select an inverse LFNST from the determined set of inverse LFNST candidates;

apply the inverse low-frequency non-separable transform (LFNST) to the first subset of dequantized coefficients to determine a first intermediate subset of coefficients; and

apply an inverse separable transform to the first intermediate subset of coefficients and at least a portion of the second subset of coefficients to determine a block of reconstructed residual values.

28. A device for decoding video data, the device comprising:

means for determining a number of allowed non-zero coefficients for a block of video data based on a size of the block;

means for obtaining a set of dequantized coefficients for the block of video data, wherein the set of dequantized coefficients comprises a first subset of dequantized coefficients that includes non-zero dequantized coefficients and a second subset of dequantized coefficients that includes all zero coefficients;

means for determining an intra prediction mode for the block of video data;

means for determining, based on the intra prediction mode, a set of inverse LFNST candidates from a plurality of sets, wherein the plurality of sets includes 35 sets, wherein the means for determining the set of inverse LFNST candidates comprises means for determining an index indicative of the set of inverse LFNST candidates from the plurality of sets based on the inter prediction mode, and the means for determining the index comprising means for determining the index to be equal to:

for values of the intra prediction mode between 0 and 34, the value of the intra prediction mode; and

for values of the intra prediction mode between 35 and 66, 68 minus the value of the intra prediction mode;

means for selecting an inverse LFNST from the determined set of inverse LFNST candidates;

means for applying the inverse low-frequency non-separable transform (LFNST) to the first subset of dequantized coefficients to determine a first intermediate subset of coefficients; and

means for applying an inverse separable transform to the first intermediate subset of coefficients and at least a portion of the second subset of coefficients to determine a block of reconstructed residual values.