Structured pruning for machine learning model

1. A method implemented by a hierarchical hardware, the method comprising:

receiving an input weight pattern of a neural network model from a computing device, and storing the input weight pattern into a weight memory of the hierarchical hardware;

obtaining an output weight pattern configured to facilitate parallel computations from the input weight pattern based on a block-wise row and column pruning method, the block-wise row and column pruning method comprising:

partitioning the input weight pattern into a mesh network of sub-patterns, each row of the input weight pattern comprising sub-rows of a first number of sub-patterns, and each column of the input weight pattern comprising sub-columns of a second number of sub-patterns;

sending each sub-pattern of the sub-patterns from the weight memory to a respective sparse group compute engine (SGCE) of a plurality of sparse group compute engines of the hierarchical hardware, the weight memory being shared and accessible by the plurality of sparse group compute engines;

pruning, by the respective SGCE, respective one or more sub-columns from each sub-pattern to achieve a predetermined column sparsity; and

pruning, by the respective SGCE, respective one or more sub-rows from each sub-pattern to achieve a predetermined row sparsity under a constraint that, for each row of the input weight pattern, at least one sub-row of a sub-pattern in a respective row of the input weight pattern is not pruned; and

sending the output weight pattern to the computing device.

2. The method of claim 1 , wherein pruning the respective one or more sub-columns comprises setting all weight values in the respective one or more sub-columns as zeros.

3. The method of claim 1 , further comprising compressing the output weight pattern in a compact form by removing sub-columns and sub-rows having weight values being all zeros from the output weight pattern.

4. The method of claim 1 , wherein pruning the respective one or more sub-columns from each sub-pattern of the mesh network of sub-patterns to achieve the predetermined column sparsity comprises: pruning a sub-column of a sub-pattern in response to determining that a norm of the sub-column of the sub-pattern is less than a first predetermined threshold.

5. The method of claim 4 , wherein the norm of the sub-column comprises a square root of a sum of each element of the sub-column being squared.

6. The method of claim 1 , wherein pruning the respective one or more sub-rows from each sub-pattern of the mesh network of sub-patterns to achieve the predetermined row sparsity under the constraint comprises: pruning a sub-row of a sub-pattern in response to determining that a norm of the sub-row of the sub-pattern is less than a second predetermined threshold.

7. The method of claim 6 , wherein pruning the respective one or more sub-rows from each sub-pattern of the mesh network of sub-patterns to achieve the predetermined row sparsity under the constraint further comprises:

determining that respective norms of all sub-rows in a certain row of the input weight pattern are less than the second predetermined threshold; and

keeping at least one sub-row having a norm that is highest among the respective norms of all the sub-rows in the certain row of the input weight pattern, and pruning remaining sub-rows in the certain sub-row of the input weight pattern.

8. The method of claim 1 , wherein pruning respective one or more sub-columns from each sub-pattern to achieve a predetermined column sparsity comprises pruning the respective one or more sub-columns from each sub-pattern so that a sparsity of sub-columns in the output weight pattern is within a predetermined range around the predetermined column sparsity.

9. One or more computer-readable media storing executable instructions that, when executed by one or more processors of a hierarchical hardware, cause the one or more processors to perform acts comprising:

receiving an input weight pattern of a neural network model from a computing device, and storing the input weight pattern into a weight memory of the hierarchical hardware;

obtaining an output weight pattern from the input weight pattern configured to facilitate parallel computations based on a block-wise row and column pruning method, the block-wise row and column pruning method comprising:

partitioning the input weight pattern into a mesh network of sub-patterns, each row of the input weight pattern comprising sub-rows of a first number of sub-patterns, and each column of the input weight pattern comprising sub-columns of a second number of sub-patterns;

sending each sub-pattern of the sub-patterns from the weight memory to a respective sparse group compute engine (SGCE) of a plurality of sparse group compute engines of the hierarchical hardware, the weight memory being shared and accessible by the plurality of sparse group compute engines;

pruning, by the respective SGCE, respective one or more sub-columns from each sub-pattern to achieve a predetermined column sparsity; and

pruning, by the respective SGCE, respective one or more sub-rows from each sub-pattern to achieve a predetermined row sparsity under a constraint that, for each row of the input weight pattern, at least one sub-row of a sub-pattern in a respective row of the input weight pattern is not pruned; and

sending the output weight pattern to the computing device.

10. The one or more computer-readable media of claim 9 , wherein pruning the respective one or more sub-columns comprises setting all weight values in the respective one or more sub-columns as zeros.

11. The one or more computer-readable media of claim 9 , the acts further comprising compressing the output weight pattern in a compact form by removing sub-columns and sub-rows having weight values being all zeros from the output weight pattern.

12. The one or more computer-readable media of claim 9 , wherein pruning the respective one or more sub-columns from each sub-pattern of the mesh network of sub-patterns to achieve the predetermined column sparsity comprises: pruning a sub-column of a sub-pattern in response to determining that a norm of the sub-column of the sub-pattern is less than a first predetermined threshold.

13. The one or more computer-readable media of claim 12 , wherein the norm of the sub-column comprises a square root of a sum of each element of the sub-column being squared.

14. The one or more computer-readable media of claim 9 , wherein pruning the respective one or more sub-rows from each sub-pattern of the mesh network of sub-patterns to achieve the predetermined row sparsity under the constraint comprises: pruning a sub-row of a sub-pattern in response to determining that a norm of the sub-row of the sub-pattern is less than a second predetermined threshold.

15. The one or more computer-readable media of claim 14 , wherein pruning the respective one or more sub-rows from each sub-pattern of the mesh network of sub-patterns to achieve the predetermined row sparsity under the constraint further comprises:

determining that respective norms of all sub-rows in a certain row of the input weight pattern are less than the second predetermined threshold; and

keeping at least one sub-row having a norm that is highest among the respective norms of all the sub-rows in the certain row of the input weight pattern, and pruning remaining sub-rows in the certain sub-row of the input weight pattern.

16. The one or more computer-readable media of claim 9 , wherein pruning respective one or more sub-columns from each sub-pattern to achieve a predetermined column sparsity comprises pruning the respective one or more sub-columns from each sub-pattern so that a sparsity of sub-columns in the output weight pattern is within a predetermined range around the predetermined column sparsity.

17. A system comprising:

a hierarchical hardware comprising: one or more neural network processing units; and memory storing executable instructions that, when executed by the one or more neural network processing units, cause the one or more neural network processing units to perform acts comprising:

receiving an input weight pattern of a neural network model from a computing device, and storing the input weight pattern into a weight memory of the hierarchical hardware;

obtaining an output weight pattern from the input weight pattern configured to facilitate parallel computations based on a block-wise row and column pruning method, the block-wise row and column pruning method comprising:

partitioning the input weight pattern into a mesh network of sub-patterns, each row of the input weight pattern comprising sub-rows of a first number of sub-patterns, and each column of the input weight pattern comprising sub-columns of a second number of sub-patterns;

sending each sub-pattern of the sub-patterns from the weight memory to a respective sparse group compute engine (SGCE) of a plurality of sparse group compute engines of the hierarchical hardware, the weight memory being shared and accessible by the plurality of sparse group compute engines;

pruning, by the respective SGCE, respective one or more sub-columns from each sub-pattern to achieve a predetermined column sparsity; and

pruning, by the respective SGCE, respective one or more sub-rows from each sub-pattern to achieve a predetermined row sparsity under a constraint that, for each row of the input weight pattern, at least one sub-row of a sub-pattern in a respective row of the input weight pattern is not pruned; and

sending the output weight pattern to the computing device.

18. The system of claim 17 , wherein pruning the respective one or more sub-columns comprises setting all weight values in the respective one or more sub-columns as zeros, and wherein the acts further comprise compressing the output weight pattern in a compact form by removing sub-columns and sub-rows having weight values being all zeros from the output weight pattern.

19. The system of claim 17 , wherein pruning the respective one or more sub-columns from each sub-pattern of the mesh network of sub-patterns to achieve the predetermined column sparsity comprises: pruning a sub-column of a sub-pattern in response to determining that a norm of the sub-column of the sub-pattern is less than a first predetermined threshold.

20. The system of claim 17 , wherein pruning the respective one or more sub-rows from each sub-pattern of the mesh network of sub-patterns to achieve the predetermined row sparsity under the constraint comprises:

pruning a sub-row of a sub-pattern in response to determining that a norm of the sub-row of the sub-pattern is less than a second predetermined threshold;

determining that respective norms of all sub-rows in a certain row of the input weight pattern are less than the second predetermined threshold; and

keeping at least one sub-row having a norm that is highest among the respective norms of all the sub-rows in the certain row of the input weight pattern, and pruning remaining sub-rows in the certain sub-row of the input weight pattern.