Accelerated sparse neural network training using transposable pruning masks
A method includes receiving a trained deep neural network (DNN) having multiple layers represented by respective weight matrices. The DNN is pruned by, for at least one layer represented by a respective weight matrix including multiple weights, converting the weight matrix into a structured sparse weight matrix by (a) defining multiple M-element line-blocks in the weight matrix, each line-block including the weights along an M-element portion of a row of the weight matrix, and respective multiple M-element column-blocks in a transpose of the weight matrix (b) specifying a subset of at least N weights in the weight matrix that are to be nulled in each M-element line-block and respective at least N weights to be nulled in each M-element column-block of the transpose of the weight matrix, and (c) nulling the weights in the subset. A deep learning operation is performed using the pruned DNN.
1 . A method, comprising:
receiving a trained deep neural network (DNN) having multiple layers represented by respective weight matrices;
generating a pruned DNN by, for at least one layer represented by a weight matrix comprising multiple weights, converting the weight matrix into a structured sparse weight matrix by:
defining in the weight matrix multiple line-blocks, each line-block comprising the weights along a portion of a row of the weight matrix, and
in each line-block, nulling a predefined subset of the weights while preserving a mean of the weights of the line-block by absorbing the mean of the predefined subset of the weights into remaining non-zero weights of the line-block, wherein each nulled weight is zero, wherein absorbing the mean of the predefined subset of the weights into the remaining non-zero weights of the line-block comprises increasing one or more absolute values of the remaining non-zero weights of the line-block based on the mean of the predefined subset of the weights; and
performing an artificial intelligence task using the pruned DNN, wherein performing the artificial intelligence task comprises performing a matrix multiplication in the pruned DNN based on the structured sparse weight matrix.
2 . The method according to claim 1 , and comprising generating, from the structured sparse weight matrix, a lower-dimension weight matrix having the nulled weights removed, and wherein the matrix multiplication is performed using the lower-dimension weight matrix.
3 . The method according to claim 1 , wherein a given line-block has at least eight weights of which at least four are nulled.
4 . The method according to claim 1 , wherein, in a given line-block, at least half of the weights are nulled.
5 . The method according to claim 1 , further comprising training the DNN.
6 . A method, comprising:
receiving a trained deep neural network (DNN) having multiple layers represented by respective weight matrices;
generating a pruned DNN by, for at least one layer represented by a weight matrix comprising multiple weights, converting the weight matrix into a structured sparse weight matrix:
finding an optimal pruning mask that (i) specifies a respective subset of the weights in the weight matrix that are to be nulled, (ii) meets a predefined pruning ratio by ensuring at least N weights out of every M successive weights are nulled, and (iii) optimizes a predefined norm,
nulling the subset of the weights specified in the optimal pruning mask by changing a value of each weight in the subset to zero, and
increasing one or more absolute values of one or more non-zero weights in the weight matrix based on the mean of the subset of the weights; and
performing an artificial intelligence task using the pruned DNN, wherein performing the artificial intelligence task comprises performing a matrix multiplication in the pruned DNN using the structured sparse weight matrix.
7 . The method according to claim 6 , and comprising generating, from the structured sparse weight matrix, a lower-dimension weight trick weight matrix having the nulled weight removed, and wherein the matrix multiplication is performed using the lower-dimension weight matrix.
8 . The method according to claim 6 , wherein M is at least eight, and N is at least four.
9 . The method according to claim 6 , wherein N is at least half of M.
10 . The method according to claim 6 , further compromising training the DNN.
11 . A method, comprising:
receiving a trained deep neural network (DNN) having multiple layers represented by respective weight matrices;
generating a pruned DNN by, for at least one layer represented by a weight matrix comprising multiple weights, converting the weight matrix into a structured sparse weight matrix by:
defining multiple M-element line-blocks in the weight matrix, each line-block comprising the weights along an M-element portion of a row of the weight matrix,
defining multiple M-element column-blocks in a transpose of the weight matrix, each column-block comprising the weights along an M-element portion of the column of the transpose of the weight matrix,
specifying a subset of the weights in the weight matrix that are to be nulled, such that (i) at least N weights in each M-element line-block of the weight matrix are to be nulled, and (ii) at least N weights in each M-element, column-block of the transpose of the weight matrix are to be nulled,
nulling the weights in the subset by changing a value of each weight in the subset to zero, and
increasing one or more absolute values of one or more non-zero weights in the weight matrix based on the mean of the subset of the weights; and
performing a deep learning operation using the pruned DNN.
12 . The method according to claim 11 , and comprising generating, from the structured sparse weight matrix, a lower-dimension weight matrix having the nulled weights removed, and wherein the deep learning operation is performed using the lower-dimension weight matrix.
13 . The method according to claim 11 , wherein a given 40 e -line-block has at least eight weights of which at least four are nulled.
14 . The method according to claim 11 , wherein, in a given line-block, at least half of the weights are nulled.
15 . An apparatus, comprising:
a memory configured to store data of a trained deep neural network open (DNN) having multiple layers representative by respective weight matrices; and
a processor configured to:
generate a pruned DNN by, for at least one layer represented by a weight matrix comprising multiple weights, converting the weight matrix into a structured sparse weight matrix by:
defining in the weight matrix multiple line-blocks, each line-block comprising the weights along a portion of a row of the weight matrix, and
in each line-block, nulling a predefined subset of the weights while preserving a mean of the weights of the line-block by absorbing the mean of the predefined subset of the weights into remaining non-zero weights of the line-block, wherein each nulled weight is zero, wherein absorbing the mean of the predefined subset of the weights into the remaining non-zero weights of the line-block comprises increasing one or more absolute values of the remaining non-zero weights of the line-block based on the mean of the predefined subset of the weights, and
perform an artificial intelligence task using the pruned DNN, wherein performing the artificial intelligence task comprises performing a matrix multiplication in a DNN using the structured sparse weight matrix.
16 . The apparatus according to claim 15 , wherein the processor is further configured to perform the artificial intelligence task by:
generating, from the structured sparse weight matrix, a lower-dimension weight matrix having the nulled weights removed; and
to performing the matrix multiplication using the lower-dimension weight matrix.
17 . The apparatus according to claim 15 , wherein a given line-block has at least eight weights of which at least four or nulled.
18 . The apparatus according to claim 15 , wherein, in a given line-block, at least half of the weights are nulled.
19 . The apparatus according to claim 15 , wherein the processor is further configured to train the DNN.
20 . An apparatus, comprising:
a memory configured to store data of a trained deep neural network (DNN) having multiple layers represented by respective weight matrices; and
a processor, which is configured to:
generate a pruned DNN by, for at least one layer represented by a weight matrix comprising multiple weights, converting the weight matrix into a structured sparse weight matrix by:
finding an optimal pruning mask that (i) specifies a respective subset of the weights in the weight matrix that are being nulled, (ii) meets a predefined pruning ratio by ensuring at least N weights out of every M successive weights are nulled, and (iii) optimizes a predefined norm,
nulling the subset of the weights specified in the optimal pruning mask by changing a value of each weight in the subset to zero, and
increasing one or more absolute values of one or more non-zero weights in the weight matrix based on the mean of the subset of the weights, and
perform an artificial intelligence task using the pruned DNN, wherein performing the artificial intelligence task comprises performing a matrix multiplication in the DNN using the structured sparse weight matrix.
21 . The apparatus according to claim 20 , wherein the processor is further configured to perform the artificial intelligence task by:
generating, from the structured sparse weight matrix, a lower-dimension weight matrix having the nulled weights removed; and
performing the matrix multiplication using the lower-dimension great matrix.
22 . The apparatus according to claim 20 , wherein M is at least eight, and N is at least four.
23 . The apparatus according to claim 20 , wherein N at least half of M.
24 . The apparatus according to claim 20 , wherein the processor is further configured to train the DNN.
25 . An apparatus, comprising:
a memory configured to store data of a trained deep neural network (DNN) having multiple layers represented by respective weight matrices; and
a processor, which is configured to:
generate a pruned DNN by, for at least one layer represented by a weight matrix comprising multiple weights, converting the weight matrix into a structured sparse weight matrix by:
defining multiple M-element line-blocks in the weight matrix, each line-block comprising the weights along an M-element portion of a row of the weight matrix,
defining multiple M-element column-blocks in a transpose of the weight matrix, each column-block comprising the weights along an M-element portion of a column of the transpose of the weight matrix,
specifying a subset of the weights in the weight matrix that are to be nulled, such that (i) at least N weights in each M-element line-block of the weight matrix are to be nulled, and (ii) at least N weights in each M-element column-block of the transpose of the weight matrix are to be nulled,
nulling the weights in the subset by changing a value of each weight in the subset to zero, and
increasing one or more absolute values of one or more non-zero weights in the weight matrix based on the mean of the subset of the weights, and
perform a deep learning operation using the pruned DNN.
26 . The apparatus according to claim 25 , where in the processor is further configured to perform the deep learning operation by:
generating, from the structured sparse weight matrix, a lower-dimension weight matrix having the nulled weights removed; and
performing the deep learning operation using the low-dimension weight matrix.
27 . The apparatus according to claim 25 , wherein a given line-block has at least eight weights of which at least four are nulled.
28 . The apparatus according to claim 25 , wherein, in a given line-block, at least half of the weights are nulled.