Granted Patent B2

US 11,825,130 · App. 18/154,245 · Granted Nov 21, 2023

Inventors: Wei Wang (Palo Alto, CA); Wei Jiang (San Jose, CA); Shan Liu (San Jose, CA)

Assignee: TENCENT AMERICA LLC

H04N19/96G06N3/08H04N19/119H04N19/13H04N19/91

Abstract

Claims (51)

1. A method of three-dimensional (3D)-Tree coding for neural network model compression, the method being performed by at least one processor, and the method comprising:

reshaping a four-dimensional (4D) parameter tensor of a neural network into a 3D parameter tensor of the neural network, the 3D parameter tensor comprising a convolution kernel size, an input feature size, and an output feature size;

partitioning the 3D parameter tensor along a plane that is formed by the input feature size and the output feature size into 3D coding tree units (CTU3Ds);

partitioning each of the CTU3Ds into a plurality of 3D coding units (CU3Ds) recursively until a predetermined depth, using a quad-tree;

constructing a 3D-Unitree for each of the plurality of CU3Ds;

constructing a 3D-Tagtree for each of the plurality of CU3Ds;

based on a value of a parent node of the 3D-Unitree being zero, encoding a corresponding unified value and skipping scanning one or more child nodes; and

based on the value of the parent node of the 3D-Unitree being non-zero, encoding the value of the parent node of the 3D-Unitree with one of a corresponding Tagtree value or a difference in the corresponding Tagtree value between the parent node and the one or more child nodes.

2. The method of claim 1 , wherein a predetermined encoding start depth indicates a first depth at which nodes are to be included in an encoding process.

3. The method of claim 2 , wherein the method comprises a predefined scan order, and wherein one or more nodes with a depth above the predetermined encoding start depth are skipped.

4. The method of claim 1 , further comprising, based on a value of a node at a lowest depth being non-zero, encoding the 3D-Unitree using a direct quantization method.

5. The method of claim 1 , further comprising, based on a value of a node at a lowest depth being non-zero, encoding the 3D-Tagtree using a direct quantization method.

6. The method of claim 1 , further comprising, based on the parent node of the 3D-Unitree having one or more absent child nodes, skipping encoding the one or more absent child nodes.

7. The method of claim 1 , wherein constructing a 3D-Unitree comprises:

based on child nodes of the parent node included in a depth other than a last depth of the 3D-Unitree having different values, the value of the parent node of the 3D-Unitree is 1,

based on the child nodes of the parent node having identical values, the value of the parent node of the 3D-Unitree is 0,

based on codebook indices or absolute values of coefficients of a respective one of the plurality of CU3Ds having different values, a value of a node included in the last depth of the 3D-Unitree is 1, and

based on the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds having identical values, the value of the node included in the last depth of the 3D-Unitree is 0.

8. An apparatus for adaptive block partitioning for neural network model compression, the apparatus comprising:

at least one memory configured to store program code; and

at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:

reshaping code configured to cause the at least one processor to reshape a four-dimensional (4D) parameter tensor of a neural network into a 3D parameter tensor of the neural network, the 3D parameter tensor comprising a convolution kernel size, an input feature size, and an output feature size;

first partitioning code configured to cause the at least one processor to partition the 3D parameter tensor along a plane that is formed by the input feature size and the output feature size, into 3D coding tree units (CTU3Ds);

second partitioning code configured to cause the at least one processor to partition each of the CTU3Ds into a plurality of 3D coding units (CU3Ds) recursively until a maximum depth, using a quad-tree;

first constructing code configured to cause the at least one processor to construct a 3D-Unitree for each of the plurality of CU3Ds;

second constructing code configured to cause the at least one processor to construct a 3D-Tagtree for each of the plurality of CU3Ds;

first encoding code configured to cause the at least one processor to, based on a value of a parent node of the 3D-Unitree being zero, encode a corresponding unified value and skipping scanning one or more child nodes; and

second encoding code configured to cause the at least one processor to, based on the value of the parent node of the 3D-Unitree being non-zero, encode the value of the parent node of the 3D-Unitree with one of a corresponding Tagtree value or a difference in the corresponding Tagtree value between the parent node and the one or more child nodes.

9. The apparatus of claim 8 , wherein a predetermined encoding start depth indicates a first depth at which nodes are to be included in an encoding process.

10. The apparatus of claim 9 , wherein the program code further comprises scanning code configured to cause the at least one processor to scan nodes in a predefined scan order, and wherein one or more nodes with a depth above the predetermined encoding start depth are skipped.

11. The apparatus of claim 8 , further comprising third encoding code configured to cause the at least one processor to, based on a value of a node at a lowest depth being non-zero, encode the 3D-Unitree using direct quantization.

12. The apparatus of claim 8 , further comprising third encoding code configured to cause the at least one processor to, based on a value of a node at a lowest depth being non-zero, encode the 3D-Tagtree using direct quantization.

13. The apparatus of claim 8 , further comprising, fourth encoding code configured to cause the at least one processor to, based on the parent node of the 3D-Unitree having one or more absent child nodes, skip encoding the one or more absent child nodes.

14. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor for adaptive block partitioning for neural network model compression, cause the at least one processor to:

reshape a four-dimensional (4D) parameter tensor of a neural network into a 3D parameter tensor of the neural network, the 3D parameter tensor comprising a convolution kernel size, an input feature size, and an output feature size;

partition the 3D parameter tensor along a plane that is formed by the input feature size and the output feature size into 3D coding tree units (CTU3Ds);

partition each of the CTU3Ds into a plurality of 3D coding units (CU3Ds) recursively until a predetermined depth, using a quad-tree;

construct a 3D-Unitree for each of the plurality of CU3Ds;

construct a 3D-Tagtree for each of the plurality of CU3Ds;

based on a value of a parent node of the 3D-Unitree being zero, encode a corresponding unified value and skipping scanning one or more child nodes; and

based on the value of the parent node of the 3D-Unitree being non-zero, encode the value of the parent node of the 3D-Unitree with one of a corresponding Tagtree value or a difference in the corresponding Tagtree value between the parent node and the one or more child nodes.

15. The non-transitory computer-readable medium of claim 14 , wherein a predetermined encoding start depth indicates a first depth at which nodes are to be included in an encoding process.

16. The non-transitory computer-readable medium of claim 15 , wherein the instructions when executed by the at least one processor, further cause the at least one processor to, scan nodes in a predefined scan order, and wherein one or more nodes with a depth above the predetermined encoding start depth are skipped.

17. The non-transitory computer-readable medium of claim 14 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to, based on a value of a node at a lowest depth being non-zero, encode the 3D-Unitree using a direct quantization method.

18. The non-transitory computer-readable medium of claim 14 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to, based on a value of a node at a lowest depth being non-zero, encode the 3D-Tagtree using a direct quantization method.

19. The non-transitory computer-readable medium of claim 14 , wherein the instructions, when executed by the at least one processor, further cause the at least one processor to, based on the parent node of the 3D-Unitree having one or more absent child nodes, skip encoding the one or more absent child nodes.

20. The non-transitory computer-readable medium of claim 14 , wherein constructing a 3D-Unitree comprises:

based on child nodes of the parent node included in a depth other than a last depth of the 3D-Unitree having different values, the value of the parent node of the 3D-Unitree is 1,

based on the child nodes of the parent node having identical values, the value of the parent node of the 3D-Unitree is 0,

based on codebook indices or absolute values of coefficients of a respective one of the plurality of CU3Ds having different values, a value of a node included in the last depth of the 3D-Unitree is 1, and

based on the codebook indices or the absolute values of the coefficients of the respective one of the plurality of CU3Ds having identical values, the value of the node included in the last depth of the 3D-Unitree is 0.

Continuity (9)