Granted Patent B2

US 11,589,078 · App. 17/536,455 · Granted Feb 21, 2023

Inventors: Wei Wang (Palo Alto, CA); Wei Jiang (San Jose, CA); Shan Liu (San Jose, CA)

Assignee: TENCENT AMERICA LLC

H04N19/96G06N3/08H04N19/119H04N19/13H04N19/91

Abstract

Claims (86)

1. A method of three-dimensional (3D)-Tree coding for neural network model compression, the method being performed by at least one processor, and the method comprising:

reshaping a four-dimensional (4D) parameter tensor of a neural network into a 3D parameter tensor of the neural network, the 3D parameter tensor comprising a convolution kernel size, an input feature size, and an output feature size;

partitioning the 3D parameter tensor along a plane that is formed by the input feature size and the output feature size into 3D coding tree units (CTU3Ds);

partitioning each of the CTU3Ds into a plurality of 3D coding units (CU3Ds) recursively until a predetermined depth, using a quad-tree; and

constructing a 3D tree for each of the plurality of CU3Ds, wherein the 3D tree for each of the plurality of CU3Ds is a 3D-Unitree,

based on child nodes of a parent node included in a depth other than a last depth of the 3D-Unitree having different values, a value of the parent node of the 3D-Unitree is 1,

based on the child nodes of the parent node having identical values, the value of the parent node of the 3D-Unitree is 0,

based on codebook indices or absolute values of coefficients of a respective one of the plurality of CU3Ds having different values, a value of a node included in the last depth of the 3D-Unitree is 1, and

based on the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds having identical values, the value of the node included in the last depth of the 3D-Unitree is 0.

2. The method of claim 1 , further comprising:

constructing a 3D-Tagtree for each of the plurality of CU3Ds,

wherein based on the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds being other than zero, a value of a node included in a last depth of the 3D-Tagtree is 1, and

based on the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds being zero, the value of the node included in the last depth of the 3D-Tagtree is 0.

3. The method of claim 2 , further comprising entropy encoding each of a plurality of values of a plurality of nodes of the 3D tree, wherein the entropy encoding comprises:

entropy encoding a value of a current node of the 3D-Unitree;

based on the entropy-encoded value of the current node of the 3D-Unitree being zero, skipping entropy encoding of values of child nodes of the current node of the 3D-Unitree;

based on the entropy-encoded value of the current node of the 3D-Unitree being non-zero and the current node of the 3D-Unitree being a top node, entropy encoding a value of a current node of the 3D-Tagtree corresponding to the current node of the 3D-Unitree; and

based on the entropy-encoded value of the current node of the 3D-Unitree being non-zero and the current node of the 3D-Unitree not being the top node, entropy encoding a difference between the value of the current node of the 3D-Tagtree and a value of a parent node of the current node of the 3D-Tagtree.

4. The method of claim 3 , wherein entropy encoding of each of the plurality of values of the plurality of nodes of the 3D tree comprises determining a start depth indicating a first depth being encoded.

5. The method of claim 4 , further comprising

encoding the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds based on the first depth being the last depth of the 3D tree or the value of the parent node at the depth other than the last depth of the 3D tree being non-zero; and

encoding a sign bit of a coefficient of the respective one of the plurality of CU3Ds based on a corresponding unified value of the respective parent node included at the depth other than the last depth of the 3D tree being non-zero and direct quantization coding method being used.

6. The method of claim 2 , further comprising

a value of a parent node at a depth other than the last depth of the 3D-Tagtree is a minimum value of child nodes of the parent node at the depth other than the last depth of the 3D-Tagtree; and

encoding a value of a last child node of the parent node at the depth other than the last depth of the 3D-Tagtree equal to the value of the parent node based on all but the last child node of the parent node at the depth other than the last depth of the 3D-Tagtree being greater than the value of the parent node.

7. The method of claim 2 , further comprising

a value of a parent node at a depth other than the last depth of the 3D-Tagtree is a maximum value of child nodes of the parent node at the depth other than the last depth of the 3D-Tagtree; and

encoding a value of a last child node of the parent node at the depth other than the last depth of the 3D-Tagtree equal to the value of the parent node based on all but the last child node of the parent node at the depth other than the last depth of the 3D-Tagtree being smaller than the value of the parent node.

8. The method of claim 1 , further comprising:

encoding a unified value of the parent node included at the depth other than the last depth of the 3D-Unitree based on determining that the value of the parent node of the 3D-Unitree is 0; and

skipping encoding of the child nodes of the parent node included at the depth other than the last depth of the 3D-Unitree.

9. An apparatus for adaptive block partitioning for neural network model compression, the apparatus comprising:

at least one memory configured to store program code; and

at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:

reshaping code configured to cause the at least one processor to reshape a four-dimensional (4D) parameter tensor of a neural network into a 3D parameter tensor of the neural network, the 3D parameter tensor comprising a convolution kernel size, an input feature size, and an output feature size;

first partitioning code configured to cause the at least one processor to partition the 3D parameter tensor along a plane that is formed by the input feature size and the output feature size, into 3D coding tree units (CTU3Ds);

second partitioning code configured to cause the at least one processor to partition each of the CTU3Ds into a plurality of 3D coding units (CU3Ds) recursively until a maximum depth, using a quad-tree; and

first constructing code configured to cause the at least one processor to construct a 3D tree for each of the plurality of CU3Ds, wherein the 3D tree for each of the plurality of CU3Ds is a 3D-Unitree,

based on child nodes of a parent node included in a depth other than a last depth of the 3D-Unitree having different values, a value of the parent node of the 3D-Unitree is 1,

based on the child nodes of the parent node having identical values, the value of the parent node of the 3D-Unitree is 0,

based on codebook indices or absolute values of coefficients of a respective one of the plurality of CU3Ds having different values, a value of a node included in the last depth of the 3D-Unitree is 1, and

based on the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds having identical values, the value of the node included in the last depth of the 3D-Unitree is 0.

10. The apparatus of claim 9 , further comprising:

second constructing code configured to cause the at least one processor to construct a 3D-Tagtree for each of the plurality of CU3Ds,

wherein based on the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds being other than zero, a value of a node included in a last depth of the 3D-Tagtree is 1, and

based on the codebook indices or the absolute values of the coefficients the respective one of the plurality of CU3Ds being zero, the value of the node included in the last depth of the 3D-Tagtree is 0.

11. The apparatus of claim 10 , further comprising first encoding code configured to cause the at least one processor to entropy encode each of a plurality of values of a plurality of nodes of the 3D tree, wherein the first encoding code comprises:

entropy encoding a value of a current node of the 3D-Unitree;

based on the entropy-encoded value of the current node of the 3D-Unitree being zero, skipping entropy encoding of values of child nodes of the current node of the 3D-Unitree;

based on the entropy-encoded value of the current node of the 3D-Unitree being non-zero and the current node of the 3D-Unitree being a top node, entropy encoding a value of a current node of the 3D-Tagtree corresponding to the current node of the 3D-Unitree; and

based on the entropy-encoded value of the current node of the 3D-Unitree being non-zero and the current node of the 3D-Unitree not being the top node, entropy encoding a difference between the value of the current node of the 3D-Tagtree and a value of a parent node of the current node of the 3D-Tagtree.

12. The apparatus of claim 11 , further comprising first determining code configured to cause the at least one processor to determine a start depth indicating a first depth being encoded.

13. The apparatus of claim 12 , further comprising:

third encoding code configured to cause the at least one processor to encode the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds based on the first depth being the last depth of the 3D tree or the value of the parent node at the depth other than the last depth of the 3D tree being non-zero; and

fourth encoding code configured to cause the at least one processor to encode a sign bit of a coefficient of the respective one of the plurality of CU3Ds based on a corresponding unified value of the respective parent node included at the depth other than the last depth of the 3D tree being non-zero and direct quantization coding method being used.

14. The apparatus of claim 10 , wherein the second constructing code further comprising:

a value of a parent node at a depth other than the last depth of the 3D-Tagtree is a minimum value of child nodes of the parent node at the depth other than the last depth of the 3D-Tagtree; and

encoding a value of a last child node of the parent node at the depth other than the last depth of the 3D-Tagtree equal to the value of the parent node based on all but the last child node of the parent node at the depth other than the last depth of the 3D-Tagtree being greater than the value of the parent node.

15. The apparatus of claim 10 , wherein the second constructing code further comprising:

a value of a parent node at a depth other than the last depth of the 3D-Tagtree is a maximum value of child nodes of the parent node at the depth other than the last depth of the 3D-Tagtree; and

encoding a value of a last child node of the parent node at the depth other than the last depth of the 3D-Tagtree equal to the value of the parent node based on all but the last child node of the parent node at the depth other than the last depth of the 3D-Tagtree being smaller than the value of the parent node.

16. The apparatus of claim 9 , further comprising:

second encoding code configured to cause the at least one processor to encode a unified value of the parent node included at the depth other than the last depth of the 3D-Unitree based on determining that the value of the parent node of the 3D-Unitree is 0; and

first skipping code configured to cause the at least one processor to skip encoding of the child nodes of the parent node included at the depth other than the last depth of the 3D-Unitree.

17. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor for adaptive block partitioning for neural network model compression, cause the at least one processor to:

reshape a four-dimensional (4D) parameter tensor of a neural network into a 3D parameter tensor of the neural network, the 3D parameter tensor comprising a convolution kernel size, an input feature size, and an output feature size;

partition the 3D parameter tensor along a plane that is formed by the input feature size and the output feature size into 3D coding tree units (CTU3Ds);

partition each of the CTU3Ds into a plurality of 3D coding units (CU3Ds) recursively until a predetermined depth, using a quad-tree; and

construct a 3D tree for each of the plurality of CU3Ds, wherein the 3D tree for each of the plurality of CU3Ds is a 3D-Unitree,

based on child nodes of a parent node included in a depth other than a last depth of the 3D-Unitree having different values, a value of the parent node of the 3D-Unitree is 1,

based on the child nodes of the parent node having identical values, the value of the parent node of the 3D-Unitree is 0,

based on codebook indices or absolute values of coefficients of a respective one of the plurality of CU3Ds having different values, a value of a node included in the last depth of the 3D-Unitree is 1, and

based on the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds having identical values, the value of the node included in the last depth of the 3D-Unitree is 0.

18. The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to:

construct a 3D-Tagtree for each of the plurality of CU3Ds,

wherein based on the codebook indices or the absolute values of coefficients of the respective one of the plurality of CU3Ds being other than zero, a value of a node included in a last depth of the 3D-Tagtree is 1, and

based on the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds being zero, the value of the node included in the last depth of the 3D-Tagtree is 0.

19. The non-transitory computer-readable medium of claim 18 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to:

entropy encode a value of a current node of the 3D-Unitree;

based on the entropy-encoded value of the current node of the 3D-Unitree being zero, skipp entropy encoding of values of child nodes of the current node of the 3D-Unitree;

based on the entropy-encoded value of the current node of the 3D-Unitree being non-zero and the current node of the 3D-Unitree being a top node, entropy encode a value of a current node of the 3D-Tagtree corresponding to the current node of the 3D-Unitree; and

based on the entropy-encoded value of the current node of the 3D-Unitree being non-zero and the current node of the 3D-Unitree not being the top node, entropy encode a difference between the value of the current node of the 3D-Tagtree and a value of a parent node of the current node of the 3D-Tagtree.

20. The non-transitory computer-readable medium of claim 19 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to:

determine a start depth indicating a first depth being encoded;

encode the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds based on the first depth being the last depth of the 3D tree or the value of the parent node at the depth other than the last depth of the 3D tree being non-zero; and

encode a sign bit of a coefficient of the respective one of the plurality of CU3Ds based on a corresponding unified value of the respective parent node included at the depth other than the last depth of the 3D tree being non-zero and direct quantization coding method being used.

Continuity (8)