Pruning and accelerating neural networks with hierarchical fine-grained structured sparsity
Hierarchical structured sparse parameter pruning and processing improves runtime performance and energy efficiency of neural networks. In contrast with conventional (non-structured) pruning which allows for any distribution of the non-zero values within a matrix that achieves the desired sparsity degree (e.g., 50%) and is consequently difficult to accelerate, structured hierarchical sparsity requires each multi-element unit at the coarsest granularity of the hierarchy to be pruned to the desired sparsity degree. The global desired sparsity degree is a function of the per-level sparsity degrees. Distribution of non-zero values within each multi-element unit is constrained according to the per-level sparsity degree at the particular level of the hierarchy. Each level of the hierarchy may be associated with a hardware (e.g., logic or circuit) structure that can be enabled or disabled according to the per-level sparsity. Hierarchical sparsity provides performance improvements for a greater variety of sparsity patterns, granularity, and sparsity degrees.
1 . A method, comprising:
pruning initial parameters of a first array for a dense neural network model according to a first level of a hierarchical structured sparsity pattern to produce pruned parameters for a single-level structured sparse neural network model corresponding to a first sparsity degree;
pruning the pruned parameters according to a second level to produce first parameters of the first array for a first hierarchical structured sparse neural network model corresponding to a second sparsity degree that is different compared with the first sparsity degree;
pruning the pruned parameters according to the second level to produce second pruned parameters for a second hierarchical structured sparse neural network model corresponding to a third sparsity degree that is different compared with the second sparsity degree;
receiving the first parameters, wherein the hierarchical structural sparsity pattern includes at least a per-element pattern corresponding to the first level that indicates positions of the first parameters within the first array and a multi-element pattern corresponding to the second level and that identifies positions of multi-elements within the first array;
for each result of an output array, selecting and processing a subset of second parameters of a second array and a corresponding subset of the first parameters according to the per-element pattern to produce intermediate results comprising sums of products; and
based on the multi-element pattern, combining the intermediate results to compute a subset of the results of the output array, wherein the combining comprises selectively disabling circuitry comprising at least a first multi-element processing unit of a plurality of multi-element processing units based on the multi-element pattern, selectively-disabling circuitry comprising at least one and less than all processing elements within a second multi-element processing unit of the plurality of multi-element processing units based on the per-element pattern, and preventing transmission of the subset of second parameters to the first multi-element processing unit.
2 . The method of claim 1 , wherein the first multi-element processing unit performs at least one of a compute operation, a memory operation, or a transfer operation.
3 . The method of claim 1 , wherein the single-level structured sparse neural network model is trained for a task.
4 . The method of claim 1 , wherein at least one of the steps of receiving, processing, and combining are performed on a server or in a data center to generate data, and the data is streamed to a user device.
5 . The method of claim 1 , wherein at least one of the steps of receiving, processing, and combining are performed within a cloud computing environment.
6 . The method of claim 1 , wherein at least one of the steps of receiving, processing, and combining are performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.
7 . The method of claim 1 , wherein at least one of the steps of receiving, processing, and combining is performed on a virtual machine comprising a portion of a graphics processing unit.
8 . The method of claim 1 , wherein a granularity of the multi-element pattern corresponds to one of multiple multi-element processing units corresponding to a number of elements stored in a cache line, a number of elements stored in a memory word, or a number of elements transmitted through a peer-to-peer connection.
9 . The method of claim 1 , wherein distribution of non-zero values within each multi-element is constrained according to the first sparsity degree and the second sparsity degree.
10 . A system, comprising:
a memory that stores a first array; and
a processor that is connected to the memory, wherein the processor comprises a plurality of multi-element processing units and is configured to:
prune initial parameters of the first array for a dense neural network model according to a first level of a hierarchical structured sparsity pattern to produce pruned parameters for a single-level structured sparse neural network model corresponding to a first sparsity degree;
prune the pruned parameters according to a second level to produce first parameters of the first array for a first hierarchical structured sparse neural network model corresponding to a second sparsity degree that is different compared with the first sparsity degree;
prune the pruned parameters according to the second level to produce second pruned parameters for a second hierarchical structured sparse neural network model corresponding to a third sparsity degree that is different compared with the second sparsity degree;
receive the first parameters, wherein the hierarchical structural sparsity pattern includes at least a per-element pattern corresponding to the first level that indicates positions of the first parameters within the first array and a multi-element pattern corresponding to the second level and that identifies positions of multi-elements within the first array;
for each result of an output array, select and process a subset of second parameters of a second array and a corresponding subset of the first parameters according to the per-element pattern to produce intermediate results comprising sums of products; and
based on the multi-element pattern, combine the intermediate results to compute a subset of the results of the output array, wherein circuitry comprising at least a first multi-element processing unit of the plurality of multi-element processing units is selectively disabled based on the multi-element pattern, circuitry comprising at least one and less than all processing elements within a second multi-element processing unit of the plurality of multi-element processing units is selectively disabled based on the per-element pattern, and transmission of the subset of second parameters to the first multi-element processing unit is prevented.
11 . The system of claim 10 , wherein the first multi-element processing unit performs at least one of a compute operation, a memory operation, or a transfer operation.
12 . The system of claim 10 , wherein the single-level structured sparse neural network model is trained for a task.
13 . The system of claim 10 , wherein a granularity of the multi-element pattern corresponds to one of multiple multi-element processing units corresponding to a number of elements stored in a cache line, a number of elements stored in a memory word, or a number of elements transmitted through a peer-to-peer connection.
14 . A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
pruning initial parameters of a first array for a dense neural network model according to a first level of a hierarchical structured sparsity pattern to produce pruned parameters for a single-level structured sparse neural network model corresponding to a first sparsity degree;
pruning the pruned parameters according to a second level to produce first parameters of the first array for a first hierarchical structured sparse neural network model corresponding to a second sparsity degree that is different compared with the first sparsity degree;
pruning the pruned parameters according to the second level to produce second pruned parameters for a second hierarchical structured sparse neural network model corresponding to a third sparsity degree that is different compared with the second sparsity degree;
receiving the first parameters, wherein the hierarchical structural sparsity pattern includes at least a per-element pattern corresponding to the first level that indicates positions of the first parameters within the first array and a multi-element pattern corresponding to the second level and that identifies positions of multi-elements within the first array;
for each result of an output array, selecting and processing a subset of second parameters of a second array and a corresponding subset of the first parameters according to the per-element pattern to produce intermediate results comprising sums of products; and
based on the multi-element pattern, combining the intermediate results to compute a subset of the results of the output array, wherein the combining comprises selectively disabling circuitry comprising at least a first multi-element processing unit of a plurality of multi-element processing units based on the multi-element pattern, selectively-disabling circuitry comprising at least one and less than all processing elements within a second multi-element processing unit of the plurality of multi-element processing units based on the per-element pattern, and preventing transmission of the subset of second parameters to the first multi-element processing unit.
15 . The non-transitory computer-readable media of claim 14 , wherein the hierarchical structural sparsity pattern further indicates an additional sparsity pattern, and further comprising based on the additional sparsity pattern, selectively disabling execution of an operation for a portion of the second parameters.