Bank-balanced-sparse activation feature maps for neural network models
Embodiments disclose bank-balanced-sparse activation neural network models and methods to generate the bank-balanced-sparse activation neural network models. According to one embodiment, a neural network sparsification engine determines a first deep neural network (DNN) model having two or more hidden layers. The engine determines a bank size, a bank layout, and a target sparsity. The engine segments the activation feature maps into a plurality of banks based on the bank size and the bank layout. The engine generates a second DNN model by increasing a sparsity for each bank of activation feature map based on the target sparsity, wherein the second DNN model is used for inferencing.
1. A method to generate a deep neural network (DNN) model, comprising:
determining a first deep neural network (DNN) model having one or more hidden layers;
determining a bank size, a bank layout, and a target sparsity, the target sparsity specifying a sparsity for a bank of an activation feature map for the one or more hidden layers;
generating a second DNN model based on the first DNN model, wherein a hidden layer of the second DNN model includes a dynamic mask to mask an activation feature map of the hidden layer; and
retraining the second DNN model by:
determining an output tensor corresponding to an activation feature map at a corresponding hidden layer of the second DNN model;
determining a plurality of banks based on the bank size and the bank layout for the output tensor, wherein the output tensor includes the plurality of banks based on the bank size and the bank layout;
computing the dynamic mask based on the output tensor;
applying the dynamic mask to the output tensor by performing component-wise multiplication between the output tensor corresponding to the activation feature map and the mask; and
increasing a sparsity for each of the plurality of banks in the output tensor until the sparsity is equal to or greater than the target sparsity while ensuring the second DNN model converges, wherein the second DNN model includes activations among the plurality of banks at the output tensor such that a number of non-zero elements in each of the plurality of banks is the same and the second DNN model is used for inferencing.
2. The method of claim 1 , wherein retraining the second DNN model comprises:
for each bank of activation feature map, increasing the sparsity for the bank of activation feature map gradually until the bank of activation feature map reaches the target sparsity.
3. The method of claim 1 , wherein determining the plurality of banks based on the bank size and the bank layout comprises:
determining the plurality of banks based on a row dimension or a column dimension of an activation tensor for a layer of the first DNN model if the activation tensor is a two-dimensional (2D) tensor, or
determining the plurality of banks based on a channel dimension, a row dimension, or a column dimension of an activation tensor for a layer of the first DNN model if the activation tensor is a three or more dimensional tensor.
4. The method of claim 2 , wherein retraining the second DNN model comprises:
calculating an output for an objective function by forward propagation; and
updating a weight parameter for each hidden layer of the second DNN model by back propagation based on the calculated output.
5. The method of claim 1 , wherein the second DNN model includes an output tensor having two banks at each row, each bank meeting the target sparsity and is balanced among the banks of each row, or wherein the second DNN model includes an output tensor having two banks at each column, each bank meeting the target sparsity and is balanced among the banks of each column.
6. The method of claim 1 , wherein the second DNN model is a dynamic DNN model, wherein weight parameters of the second DNN model for a first inference is different than weight parameters for a second inference.
7. The method of claim 2 , wherein the sparsity for each bank of activation feature map is increased gradually by retaining top-k values from each bank and zeroing remaining values gradually for the bank, wherein k is equal to or greater than 1.
8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
determining a first deep neural network (DNN) model having one or more hidden layers;
determining a bank size, a bank layout, and a target sparsity, the target sparsity specifying a sparsity for a bank of an activation feature map for the one or more hidden layers;
generating a second DNN model based on the first DNN model, wherein a hidden layer of the second DNN model includes a dynamic mask to mask an activation feature map of the hidden layer; and
retraining the second DNN model by:
determining an output tensor corresponding to an activation feature map at a corresponding hidden layer of the second DNN model;
determining a plurality of banks based on the bank size and the bank layout for the output tensor, wherein the output tensor includes the plurality of banks based on the bank size and the bank layout;
computing the dynamic mask based on the output tensor;
applying the dynamic mask to the output tensor by performing component-wise multiplication between the output tensor corresponding to the activation feature map and the mask; and
increasing a sparsity for each of the plurality of banks in the output tensor until the sparsity is equal to or greater than the target sparsity while ensuring the second DNN model converges, wherein the second DNN model includes activations among the plurality of banks at the output tensor such that a number of non-zero elements in each of the plurality of banks is the same and the second DNN model is used for inferencing.
9. The non-transitory machine-readable medium of claim 8 , wherein retraining the second DNN model comprises:
for each bank of activation feature map, increasing the sparsity for the bank of activation feature map gradually until the bank of activation feature map reaches the target sparsity.
10. The non-transitory machine-readable medium of claim 8 , wherein determining the plurality of banks based on the bank size and the bank layout comprises:
determining the plurality of banks based on a row dimension or a column dimension of an activation tensor for a layer of the first DNN model if the activation tensor is a two dimensional (2D) tensor, or
determining the plurality of banks based on a channel dimension, a row dimension, or a column dimension of an activation tensor for a layer of the first DNN model if the activation tensor is a three or more dimensional tensor.
11. The non-transitory machine-readable medium of claim 9 , wherein retraining the second DNN model comprises:
calculating an output for an objective function by forward propagation; and
updating a weight parameter for each hidden layer of the second DNN model by back propagation based on the calculated output.
12. The non-transitory machine-readable medium of claim 8 , wherein the second DNN model includes an output tensor having two banks at each row, each bank meeting the target sparsity and is balanced among the banks of each row, or wherein the second DNN model includes an output tensor having two banks at each column, each bank meeting the target sparsity and is balanced among the banks of each column.
13. The non-transitory machine-readable medium of claim 8 , wherein second DNN model is a dynamic DNN model, wherein weight parameters of the second DNN model for a first inference is different than weight parameters for a second inference.
14. The non-transitory machine-readable medium of claim 9 , wherein the sparsity for each bank of activation feature map is increased gradually by retaining k top values from each bank and zeroing remaining values gradually for the bank, wherein k is equal to or greater than 1.
15. A computing apparatus comprising a memory having code stored therein, the code comprising: a bank-balanced-sparse deep neural network (DNN) model, the DNN model comprising:
an input layer;
one or more hidden layers; and
an output layer, wherein the one or more hidden layers are disposed between the input layer and the output layer, and each hidden layer is to output an activation feature map,
wherein the bank-balanced-sparse DNN model is generated by
determining a first DNN model having one or more hidden layers,
determining a bank size, a bank layout, and a target sparsity, the target sparsity specifying a sparsity for a bank of an activation feature map for the one or more hidden layers,
generating a second DNN model based on the first DNN model, wherein a hidden layer of the second DNN model includes a dynamic mask to mask an activation feature map of the hidden layer; and
retraining the second DNN model as the bank-balanced-sparse DNN model by:
determining an output tensor corresponding to an activation feature map at a corresponding hidden layer of the second DNN model,
determining a plurality of banks based on the bank size and the bank layout for the output tensor, wherein the output tensor includes the plurality of banks based on the bank size and the bank layout,
computing the dynamic mask based on the output tensor, and
applying the dynamic mask to the output tensor by performing component-wise multiplication between the output tensor corresponding to the activation feature map and the mask; and
increasing a sparsity for each of the plurality of banks in the output tensor until the sparsity is equal to or greater than the target sparsity while ensuring the second DNN model converges, wherein the second DNN model includes activations among the plurality of banks at the output tensor such that a number of non-zero elements in each of the plurality of banks is the same and the second DNN model is used for inferencing.
16. The apparatus of claim 15 , wherein retraining the second DNN model comprises:
for each bank of activation feature map, increasing the sparsity for the bank of activation feature map gradually until the bank of activation feature map reaches the target sparsity.
17. The apparatus of claim 15 , wherein determining the plurality of banks based on the bank size and the bank layout comprises:
determining the plurality of banks based on a row dimension or a column dimension of an activation tensor for a layer of the first DNN model if the activation tensor is a two dimensional (2D) tensor, or
determining the plurality of banks based on a channel dimension, a row dimension, or a column dimension of an activation tensor for a layer of the first DNN model if the activation tensor is a three or more dimensional tensor.
18. The apparatusbank of claim 16 , wherein retraining the second DNN model comprises:
calculating an output for an objective function by forward propagation; and
updating a weight parameter for each hidden layer of the second DNN model by back propagation based on the calculated output.
19. The apparatus of claim 15 , wherein the second DNN model includes an output tensor having two banks at each row, each bank meeting the target sparsity and is balanced among the banks of each row, or wherein the second DNN model includes an output tensor having two banks at each column, each bank meeting the target sparsity and is balanced among the banks of each column.
20. The apparatus of claim 15 , wherein second DNN model is a dynamic DNN model, wherein weight parameters of the second DNN model for a first inference is different than weight parameters for a second inference.
21. The apparatus of claim 15 , wherein the sparsity for each bank of activation feature map is increased gradually by retaining k top values from each bank and zeroing remaining values gradually for the bank, wherein k is equal to or greater than 1.