COMPRESSION METHOD OF DEEP NEURAL NETWORKS
The present disclosure proposes an improved compression method for neural networks (e.g. LSTM), which may effectively shorten the training period of a neural network by combining pruning operation into the training process, so as to reduce the number of iteration in the training process.
1 . (canceled)
2 . (canceled)
3 . (canceled)
4 . (canceled)
5 . (canceled)
6 . (canceled)
7 . (canceled)
8 . (canceled)
9 . (canceled)
10 . (canceled)
11 . (canceled)
12 . (canceled)
13 . (canceled)
14 . (canceled)
15 . (canceled)
16 . (canceled)
17 . A method for configuring a computer system comprising a network of processors, the network comprising a set of first processors and a set of second processors, wherein outputs of the first processors are coupled to outputs of the second processors; the method comprising:
predetermining a first fraction of reduction of coupling of the outputs of the first processors to the outputs of the second processors;
adjusting the network by reducing the coupling, by the first fraction of reduction;
predetermining a second fraction of reduction of the coupling of the outputs of the first processors to the outputs of the second processors;
adjusting the network by further reducing the coupling, by the second fraction of reduction;
generating a display based on the outputs of the first processors after the network is adjusted.
18 . The method of claim 17 , wherein predetermining the first fraction of reduction is based on a first target amount of the coupling.
19 . The method of claim 18 , wherein the first target amount is a function of a final target amount of the coupling.
20 . The method of claim 17 , wherein predetermining the second fraction of reduction is based on a second target amount of the coupling.
21 . The method of claim 20 , wherein predetermining the first fraction of reduction is based on a first target amount of the coupling; and wherein the second target amount equals the first target amount.
22 . The method of claim 20 , wherein predetermining the first fraction of reduction is based on a first target amount of the coupling; and wherein the second target amount is less than the first target amount.
23 . The method of claim 20 , wherein predetermining the first fraction of reduction is based on a first target amount of the coupling; and wherein predetermining the second fraction of reduction is based on the first target amount.
24 . The method of claim 19 , further comprising obtaining the final target amount of the coupling based on a relationship between the coupling and a word error ratio (WER) of the network.
25 . The method of claim 17 , wherein reducing the coupling comprises ranking strengths of coupling between pairs of the outputs of the first processors and the outputs of the second processors.
26 . The method of claim 17 , further comprising: after reducing the coupling, adjusting the network by further adjusting the coupling.
27 . The method of claim 26 , wherein further adjusting the coupling is based on a set of training data.
28 . The method of claim 17 , further comprising:
obtaining a first constraint of a distribution of non-zero coupling between pairs of the outputs of the first processors and the outputs of the second processors;
wherein reducing the coupling by the first fraction of reduction is subject to the first constraint.
29 . The method of claim 17 , further comprising:
obtaining a second constraint of a distribution of non-zero coupling between pairs of the outputs of the first processors and the outputs of the second processors;
wherein further reducing the coupling by the second fraction of reduction is subject to the second constraint.
30 . The method of claim 29 , further comprising:
obtaining a first constraint of a distribution of non-zero coupling between pairs of the outputs of the first processors and the outputs of the second processors;
wherein reducing the coupling by the first fraction of reduction is subject to the first constraint; and
wherein the first constraint and the second constraint are different.
31 . A computer program product comprising a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing a method for configuring a computer system comprising a network of processors, the network comprising a set of first processors and a set of second processors, wherein outputs of the first processors are coupled to outputs of the second processors;
the method comprising:
predetermining a first fraction of reduction of coupling of the outputs of the first processors to the outputs of the second processors;
adjusting the network by reducing the coupling, by the first fraction of reduction;
predetermining a second fraction of reduction of the coupling of the outputs of the first processors to the outputs of the second processors;
adjusting the network by further reducing the coupling, by the second fraction of reduction;
generating a display based on the outputs of the first processors after the network is adjusted.