Compression method for deep neural networks with load balance

1. A method for compressing a neural network, wherein connection relations between neurons of said neural network are characterized by a plurality of matrices, said method comprising:

dividing at least one of said plurality of matrices into a plurality of submatrices;

compressing each of the plurality of submatrices into a corresponding sparse submatrix according to a compression ratio common to all of the plurality of submatrices divided from a same matrix, including, for each submatrix of the plurality of submatrices divided from the same matrix:

setting weight values of M elements of all elements of the submatrix to zero, wherein M is a positive integer, wherein M divided by a number of all the elements of the submatrix is equal to the compression ratio, and wherein if all the elements of the submatrix are arranged in a list from top to bottom in order of decreasing magnitude of weight value, then said M elements are the bottom M elements in the list; and

encoding said compressed sparse submatrices to generate encoded compressed sparse submatrices,

wherein the compression ratio is determined by:

obtaining an initial word error ratio (WER initial ) of the neural network;

obtaining the word error ratios WER d1 , WER d2 . . . WER dn of the neural network compressed according to different compression ratios d 1 , d 2 , . . . d n , respectively; and

selecting the compression ratio from said different compression ratios d 1 , d 2 , d n based on said word error ratios WER d1 , WER d2 . . . WER dn of the compressed neural network, and

wherein said selecting the compression ratio from said different compression ratios includes:

calculating respective ΔWERs, which are the differences between WER initial and WER d1 , WER d2 , . . . , WER dn ; and

selecting the maximum compression ratio among all the compression ratios with a ΔWER below a predetermined threshold.

2. The method of claim 1 , wherein dividing the at least one of said plurality of matrices into the plurality of submatrices further includes:

dividing said matrix by regularly extracting one row out of every N rows and assigning such extracted rows to one submatrix, so as to form N submatrices.

3. The method of claim 2 , wherein said N is determined by the number of processing elements used for running the neural network, and said processing elements compute in a parallel manner.

4. The method of claim 1 , wherein encoding said compressed sparse submatrices further includes:

encoding said compressed sparse submatrices using the Compressed Row Storage (CRS) or Compressed Column Storage (CCS) format.

5. The method according to claim 1 , further includes:

providing the encoded compressed sparse submatrices to a plurality of parallel processing elements.

6. A method for compressing a neural network, wherein connection relations between neurons of said neural network are characterized by a plurality of matrices, said method comprising:

dividing at least one of said plurality of matrices into a plurality of submatrices;

analyzing sensitivity of each of said plurality of matrices, so as to determine a respective initial compression ratio for each of said plurality of matrices;

compressing the plurality of submatrices of the at least one of said plurality of matrices according to its corresponding initial compression ratio, including, for each submatrix of the all submatrices divided from the same matrix:

setting weight values of M elements of all elements of the submatrix to zero, wherein M is a positive integer, wherein M divided by a number of all the elements of the submatrix is equal to the corresponding initial compression ratio, and wherein if all the elements of the submatrix are arranged in a list from top to bottom in order of decreasing magnitude of weight value, then said M elements are the bottom M elements in the list; and

fine-tuning said compressed neural network, so as to obtain a final neural network,

wherein said analyzing the sensitivity of each of said plurality of matrices includes:

obtaining an initial word error ratio (WER initial ) of the neural network;

obtaining the word error ratios WER d1 WER d2 . . . WER dn of the neural network compressed according to different compression ratios d 1 , d 2 , . . . d n , respectively; and

selecting the initial compression ratio from said different compression ratios d 1 , d 2 , . . . d n based on said word error ratios WER d1 , WER d2 . . . WER dn of the compressed neural network, and

wherein said selecting the initial compression ratio from said different compression ratios includes:

calculating respective ΔWERs, which are the differences between WER initial and WER d1 , WER d2 , . . . , WER dn ; and

selecting the maximum compression ratio among all the compression ratios with a ΔWER below a predetermined threshold.

7. The method of claim 6 , further includes:

repeating said dividing at least one of said plurality of matrices into a plurality of submatrices, said analyzing the sensitivity of each of said plurality of matrices, said compressing the plurality of submatrices of respective matrix according to its corresponding initial compression ratio and said fine-tuning said compressed neural network iteratively.

8. The method of claim 6 , wherein said neural network is LSTM neural network.

9. The method of claim 6 , wherein dividing the at least one of said plurality of matrices into the plurality of submatrices further includes:

dividing each matrix by regularly extracting one row out of every N rows and assigning such extracted rows to one submatrix, so as to form N submatrices.

10. The method of claim 9 , wherein said N is determined by the number of processing elements used for running the neural network, and said processing elements compute in a parallel manner.

11. The method of claim 6 , wherein compressing the plurality of submatrices of the respective matrix according to its corresponding initial compression ratio further includes:

adjusting said initial compression ratio of each submatrix based on the word error ratio (WER) of the neural network obtained after compressing each submatrix based on its initial compression ratio, so as to obtain an adjusted compression ratio for each submatrix;

compressing each submatrix based on corresponding adjusted compression ratio, so as to obtain said compressed neural network.

12. The method according to claim 6 , wherein fine-tuning said compressed neural network further includes:

training said compressed neural network using a training dataset and updating the weight matrix;

determining whether said weight matrix has converged;

if not, returning to said training step; and

if yes, regarding the resultant neural network as said final neural network.

13. The method according to claim 12 , wherein training said compressed neural network using the training dataset and updating the weight matrix further includes:

inputting a training dataset and calculating the derivative of a loss function for the neural network so as to obtain a gradient matrix;

updating the weight matrix of the neural network, wherein updated weight matrix=weight matrix before updating learning rate*gradient matrix;

calculating the loss of the neural network based on said updated weight matrix;

deciding whether the present iteration is valid, wherein if the loss of the present iteration is greater than that of the previous iteration, then the present iteration is invalid; and

if the loss of the present iteration is smaller than that of the previous iteration, the present iteration is valid;

if the present iteration is invalid and the total number of iterations does not reach its maximum, adjusting the learning rate and repeating training said compressed neural network based on the previous valid iteration;

if the present iteration is valid, determining whether said weight matrix has converged.

14. A method for compressing a neural network, wherein weights between neurons of said neural network are characterized by a plurality of matrices, said method comprising:

analyzing sensitivity of each of said plurality of matrices, and determining an initial compression ratio for each of said plurality of matrices;

compressing each of said plurality of matrices based on said initial compression ratio, so as to obtain a compressed neural network, including

dividing each of said plurality of matrices into a plurality of submatrices;

pruning said submatrices into sparse submatrices according to said initial compression ratio, including, for each submatrix of the all submatrices divided from the same matrix:

setting weight values of M elements of all elements of the submatrix to zero, wherein M is a positive integer, wherein M divided by a number of all the elements of the submatrix is equal to said initial compression ratio, and wherein if all the elements of the submatrix are arranged in a list from top to bottom in order of decreasing magnitude of weight value, then said M elements are the bottom M elements in the list; and

fine-tuning said compressed neural network,

wherein said analyzing the sensitivity of each of said plurality of matrices and determining the initial compression ratio for each of said plurality of matrices includes:

obtaining an initial word error ratio (WER initial ) of the neural network;

obtaining the word error ratios WER d1 , WER d2 . . . WER dn of the neural network compressed according to different compression ratios d 1 , d 2 , . . . d n , respectively; and

selecting the initial compression ratio from said different compression ratios d 1 , d 2 , . . . d n based on said word error ratios WER d1 , WER d2 . . . WER dn of the compressed neural network, and

wherein said selecting the initial compression ratio from said different compression ratios includes:

calculating respective ΔWERs, which are the differences between WER initial and WER d1 , WER d2 , . . . WER dn ; and

selecting the maximum compression ratio among all the compression ratios with a ΔWER below a predetermined threshold.

15. The method of claim 14 , further includes:

repeating said analyzing the sensitivity of each of said plurality of matrices, said compressing each of said plurality of matrices based on said initial compression ratio and said fine-tuning said compressed neural network in at least one iteration.

16. The method of claim 14 , wherein dividing each of said plurality of matrices into the plurality of submatrices further includes:

dividing each matrix by regularly extracting one row out of every N rows and assigning such extracted rows to one submatrix, so as to form N submatrices.