Compression of deep neural networks with proper use of mask
The present invention relates to artificial neural networks, for example, deep neural networks. In particular, the present invention relates to a compression method for deep neural networks with proper use of mask and the device thereof. More specifically, the present invention relates to how to compress dense neural networks into sparse neural networks while maintaining or even improving the accuracy of the neural networks after compression.
1. A method for compressing a neural network nnet 0 with a mask matrix, said neural network is characterized by a plurality of matrices, said method comprising:
mask matrix obtaining step of obtaining a mask matrix M, wherein said mask matrix M records the distribution of non-zero elements in a plurality of sparse matrices of the neural network;
first pruning step of pruning the neural network nnet 0 using said mask matrix M to obtain a pruned neural network nnet i 1 ;
first retraining step of retraining the pruned neural network nnet i 1 that has a shape of the mask matrix without a mask, wherein the pruned neural network nnet i 1 is retrained without using said mask matrix M to obtain a dense neural network nnet o 1 ;
second pruning step of pruning said dense neural network nnet o 1 using said mask matrix M to obtain a sparse neural network nnet i 2 ;
second retraining step of retraining said sparse neural network nnet i 2 using said mask matrix M to obtain a final neural network nnet o 2 .
2. The method of claim 1 , wherein said mask matrix obtaining step further includes:
pruning said neural network to be compressed nnet 0 to obtain a plurality of sparse matrices containing zero and non-zero elements;
obtaining said mask matrix M by using elements “1” to represent non-zero elements in corresponding sparse matrix.
3. The method of claim 1 , wherein said first pruning step further includes:
point multiplying said neural network to be compressed nnet 0 with said mask matrix M to obtain said pruned neural network nnet i 1 : nnet i 1 =M ⊚ nnet 0 .
4. The method of claim 1 , wherein said first retraining step further includes:
training step, for training said pruned neural network nnet i 1 using a training dataset and updating a weight matrix of the neural network to generate an updated weight matrix without using said mask matrix M;
deciding step, for determining whether the updated weight matrix has converged;
if not, returning to said training step; and
if yes, the resultant neural network is regarded as said dense neural network nnet o 1 .
5. The method according to claim 4 , wherein said training step further includes:
inputting said training dataset and calculating the derivative of a loss function for the neural network so as to obtain a gradient matrix;
updating the weight matrix of the neural network, wherein the updated weight matrix=weight matrix before updating− learning rate*gradient matrix;
calculating the loss of the neural network based on said updated weight matrix;
deciding whether the present iteration is valid, wherein if the loss of the present iteration is greater than that of the previous iteration, then the present iteration is invalid; and
if the loss of the present iteration is smaller than that of the previous iteration, the present iteration is valid;
if the present iteration is invalid and the total number of iterations does not reach its maximum, adjusting the learning rate and repeating the training step based on the previous valid iteration;
if the present iteration is valid, repeating said deciding step for determining whether said updated weight matrix has converged.
6. The method of claim 1 , wherein said second pruning step further includes:
point multiplying said dense neural network nnet o 1 with said mask matrix M to obtain said sparse neural network nnet i 2 : nnet i 2 =M ⊚ nnet 0 1 .
7. The method of claim 1 , wherein said second retraining step further includes:
training step, for training said sparse neural network nnet i 2 using a training dataset and updating a weight matrix of the neural network to generate an updated weight matrix using said mask matrix M;
deciding step, for determining whether the updated weight matrix has converged;
if not, returning to said training step; and
if yes, the resultant neural network is regarded as said final neural network nnet o 2 .
8. The method according to claim 7 , wherein said training step further includes:
inputting said training dataset and calculating the derivative of a loss function for the neural network so as to obtain a gradient matrix;
updating the weight matrix of the neural network, wherein the updated weight matrix=weight matrix before updating− learning rate*gradient matrix*said mask matrix;
calculating the loss of the neural network based on said updated weight matrix;
deciding whether the present iteration is valid, wherein if the loss of the present iteration is greater than that of the previous iteration, then the present iteration is invalid; and
if the loss of the present iteration is smaller than that of the previous iteration, the present iteration is valid;
if the present iteration is invalid and the total number of iterations does not reach its maximum, adjusting the learning rate and repeating the training step based on the previous valid iteration;
if the present iteration is valid, repeating said deciding step for determining whether said updated weight matrix has converged.
9. The method according to claim 1 , wherein said mask matrix M contains elements “0” and “1”, wherein:
elements “1” mean that elements in corresponding position of a weight matrix of the neural network are remained, while elements “0” mean that elements in corresponding position of the weight matrix are set to zero.