Extended low-frequency non-separable transform (LFNST) designs with worst-case complexity handling
A video decoder can be configured to determine a number of allowed non-zero coefficients for a block of video data based on a size of the block; obtain a set of dequantized coefficients for the block, wherein the set of dequantized coefficients comprises a first subset of dequantized coefficients that includes non-zero dequantized coefficients and a second subset of dequantized coefficients that includes all zero coefficients, wherein a number of coefficients in the first subset of dequantized coefficients is equal to the number of allowed non-zero coefficients for the block of video data; apply an inverse low-frequency non-separable transform (LFNST) to the first subset of dequantized coefficients to determine a first intermediate subset of coefficients; and apply an inverse separable transform to the first intermediate subset of coefficients and at least a portion of the second subset of coefficients to determine a block of reconstructed residual values.
1 . A method of decoding video data, the method comprising:
determining a number of allowed non-zero coefficients for a block of reconstructed values according to the equation NZ=floor((WCM×nTbW×nTbH)/(TD1×TD2)), wherein nTbH represents a height of the block, nTbW represents a width of the block, NZ equals the number of allowed non-zero coefficients, TD1 represents a first dimension of an inverse low-frequency non-separable transform (LFNST), TD2 represents a second dimension of the LFNST, and WCM represents a constant value;
obtaining a set of dequantized coefficients for the block of video data, wherein the set of dequantized coefficients comprises a first subset of dequantized coefficients that includes non-zero dequantized coefficients and a second subset of dequantized coefficients that includes all zero coefficients, wherein a number of coefficients in the first subset of dequantized coefficients is equal to the number of allowed non-zero coefficients for the block of video data;
applying an inverse low-frequency non-separable transform (LFNST) to the first subset of dequantized coefficients to determine a first intermediate subset of coefficients; and
applying an inverse separable transform to the first intermediate subset of coefficients and at least a portion of the second subset of coefficients to determine a block of reconstructed residual values.
2 . The method of claim 1 , further comprising:
determining an intra prediction mode for the block of video data;
based on the intra prediction mode, determining a set of inverse LFNST candidates from a plurality of sets; and
selecting the inverse LFNST from the determined set of inverse LFNST candidates.
3 . The method of claim 2 , wherein the set of inverse LFNST candidates includes 3 candidates.
4 . The method of claim 2 , wherein the plurality of sets includes 35 sets.
5 . The method of claim 1 , wherein the first subset of dequantized coefficients comprises 16 dequantized coefficients, and wherein first intermediate subset of coefficients comprises 64 coefficients.
6 . The method of claim 1 , wherein the first subset of dequantized coefficients comprises 64 dequantized coefficients, and wherein first intermediate subset of coefficients comprises 64 coefficients.
7 . The method of claim 1 , wherein the first subset of dequantized coefficients comprises NZ dequantized coefficients.
8 . The method of claim 1 , wherein the first subset of dequantized coefficients comprises 2×NZ dequantized coefficients, and wherein first intermediate subset of coefficients comprises 64 coefficients.
9 . The method of claim 1 , further comprising:
adding the block of reconstructed residual values to a prediction block to form a reconstructed block;
applying one or more filters to the reconstructed block to determine a filtered reconstructed block; and
outputting decoded video data that includes the filtered reconstructed block.
10 . The method of claim 1 , wherein the method is performed as part of a video encoding process.
11 . A device for decoding video data, the device comprising:
a memory configured to store video data;
one or more processors implemented in circuitry and configured to:
determine a number of allowed non-zero coefficients for a block of reconstructed values according to the equation NZ=floor((WCM×nTbW×nTbH)/(TD1×TD2)), wherein nTbH represents a height of the block, nTbW represents a width of the block, NZ equals the number of allowed non-zero coefficients, TD1 represents a first dimension of an inverse low-frequency non-separable transform (LFNST), TD2 represents a second dimension of the LFNST, and WCM represents a constant value;
obtain a set of dequantized coefficients for the block of video data, wherein the set of dequantized coefficients comprises a first subset of dequantized coefficients that includes non-zero dequantized coefficients and a second subset of dequantized coefficients that includes all zero coefficients, wherein a number of coefficients in the first subset of dequantized coefficients is equal to the number of allowed non-zero coefficients for the block of video data;
apply an inverse low-frequency non-separable transform (LFNST) to the first subset of dequantized coefficients to determine a first intermediate subset of coefficients; and
apply an inverse separable transform to the first intermediate subset of coefficients and at least a portion of the second subset of coefficients to determine a block of reconstructed residual values.
12 . The device of claim 11 , wherein the one or more processors are further configured to:
determine an intra prediction mode for the block of video data;
based on the intra prediction mode, determine a set of inverse LFNST candidates from a plurality of sets; and
select the inverse LFNST from the determined set of inverse LFNST candidates.
13 . The device of claim 12 , wherein the set of inverse LFNST candidates includes 3 candidates.
14 . The device of claim 12 , wherein the plurality of sets includes 35 sets.
15 . The device of claim 11 , wherein the first subset of dequantized coefficients comprises 16 dequantized coefficients, and wherein first intermediate subset of coefficients comprises 64 coefficients.
16 . The device of claim 11 , wherein the first subset of dequantized coefficients comprises 64 dequantized coefficients, and wherein first intermediate subset of coefficients comprises 64 coefficients.
17 . The device of claim 11 , wherein the first subset of dequantized coefficients comprises NZ dequantized coefficients.
18 . The device of claim 11 , wherein the first subset of dequantized coefficients comprises 2×NZ dequantized coefficients, and wherein first intermediate subset of coefficients comprises 64 coefficients.
19 . The device of claim 11 , wherein the one or more processors are further configured to:
add the block of reconstructed residual values to a prediction block to form a reconstructed block;
apply one or more filters to the reconstructed block to determine a filtered reconstructed block; and
output decoded video data that includes the filtered reconstructed block.
20 . The device of claim 11 , wherein the device comprises a video encoder.
21 . The device of claim 11 , wherein the device comprises a wireless communication device, further comprising a receiver configured to receive encoded video data.
22 . The device of claim 21 , wherein the wireless communication device comprises a telephone handset and wherein the receiver is configured to demodulate, according to a wireless communication standard, a signal comprising the encoded video data.
23 . The device of claim 11 , further comprising:
a display configured to display decoded video data.
24 . The device of claim 11 , wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.
25 . The device of claim 11 , further comprising:
a camera configured to capture video data.
26 . The device of claim 11 , wherein the device comprises a wireless communication device, further comprising a transmitter configured to transmit encoded video data.
27 . The device of claim 26 , wherein the wireless communication device comprises a telephone handset and wherein the transmitter is configured to modulate, according to a wireless communication standard, a signal comprising the encoded video data.