Method and apparatus for pruning neural networks
The present invention relates to a method for pruning a neural network comprising a plurality of neurons, said method comprising: an initialization phase, wherein input information is fetched comprising at least parameters ({w ni ,b ni }) related to said neural network and a dataset (D) representative of a task that said neural network has to deal with, wherein said parameters ({w ni ,b ni }) comprising a weights vector (w ni ) and/or a bias (b ni ) related to at least one neuron of said plurality of neurons; a regularization phase, wherein said neural network is trained according to a training algorithm by using said dataset (D); a thresholding phase, wherein an element (w nij ) of said weights vector (w ni ) is put at zero when its absolute value is below a given threshold (T).
1 . A method of pruning a neural network comprising a plurality of neurons, said method comprising:
an initialization phase, wherein input information is fetched, said input information comprising at least parameters ({w ni ,b ni }) related to said neural network and a dataset (D) representative of a task that said neural network has to deal with, wherein said parameters ({w ni ,b ni }) comprise one or more of a weights vector (w ni ) or a bias (b ni ) related to at least one neuron of said plurality of neurons;
a regularization phase, wherein said neural network is trained according to a training algorithm by using said dataset (D);
a thresholding phase, wherein an element (w ni ) of said weights vector (w ni ) is put at zero when its absolute value is below a given threshold (7),
said method being characterized in that, during said regularization phase, said parameters ({w ni ,b ni }) evolve according to a regularized update rule based on a neural sensitivity measure (S) such that the neural network is pruned by being driven towards zero parameters related to at least one less sensitive neuron of said neural network, wherein said neural sensitivity measure (S) is based on a pre-activation signal of at least one neuron of said plurality of neurons.
2 . The method of claim 1 , wherein said pre-activation signal depends on an input values vector (Y n-1 ) and the parameters ({w ni ,b ni }) of said at least one neuron.
3 . The method of claim 1 , wherein said regularized update rule drives toward zero the weights vector (w ni ) related to said at least one less sensitive neuron.
4 . The method of claim 1 , wherein said neural sensitivity measure (S) represents a measure of the global impact of a given neuron in an output layer of said neural network, wherein all the previous neurons are involved in the computation of said neural sensitivity measure (S).
5 . The method of claim 1 , wherein said neural sensitivity measure (S) is based only on said pre-activation signal and an output signal of each of said neurons individually.
6 . The method of claim 1 , wherein the value of said threshold (T) is determined by increasing it from a minimum value (T min ) until an optimal value (T*) is reached, wherein said minimum value (T min ) is determined by taking the minimum of the entire set of said parameters ({w ni ,b ni }).
7 . The method of claim 1 , wherein said bias (b ni ) related to at least one neuron is put at zero when its corresponding weights vector (w ni ) is put at zero.
8 . The method of claim 1 , wherein said training algorithm is the Gradient Descent algorithm or the Stochastic Gradient Descent algorithm.
9 . The method of claim 1 , further comprising an output phase wherein processed data, such as pruned parameters ({w* ni ,b* ni }) and/or pruned model (M*) of a resulting pruned neural network, are provided as output.
10 . The method of claim 9 , wherein said processed data are transmitted to a device.
11 . An apparatus for pruning a neural network comprising a plurality of neurons, said apparatus comprising a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform operations, the operations comprising:
an initialization phase, wherein input information is fetched and said input information comprises at least parameters ({w ni ,b ni }) related to said neural network, a dataset (D) representative of a task that said neural network has to deal with, wherein said parameters ({w ni ,b ni }) comprise one or more of a weights vector (w ni ) or a bias (b ni ) related to at least one neuron of said plurality of neurons;
a regularization phase, wherein said neural network is trained according to a training algorithm by using said dataset (D);
a thresholding phase, wherein an element (w ni ) of said weights vector (w ni ) is put at zero when its absolute value is below a given threshold (T),
said apparatus being characterized in that, during said regularization phase, said parameters ({w ni ,b ni }) evolve according to a regularized update rule based on a neural sensitivity measure (S) such that the neural network is pruned by being driven towards zero parameters related to at least one less sensitive neuron of said neural network, wherein said neural sensitivity measure (S) is based on a pre-activation signal of at least one neuron of said plurality of neurons.
12 . The apparatus of claim 11 , wherein said pre-activation signal depends on an input values vector (Y n-1 ) and the parameters ({w ni ,b ni }) of said at least one neuron.
13 . The apparatus of claim 11 , wherein said regularized update rule drives toward zero the weights vector (w ni ) related to said at least one less sensitive neuron.
14 . The apparatus of claim 11 , wherein said neural sensitivity measure (S) represents a measure of the global impact of a given neuron in an output layer of said neural network, wherein all the previous neurons are involved in the computation of said neural sensitivity measure (S).
15 . The apparatus of claim 11 , wherein said neural sensitivity measure (S) is based only on said pre-activation signal and an output signal of each of said neuron individually.
16 . The apparatus of claim 11 , wherein the value of said threshold (T) is determined by increasing it from a minimum value (T min ) until an optimal value (T*) is reached, wherein said minimum value (T min ) is determined by taking the minimum of the entire set of said parameters ({w ni ,b ni }).
17 . The apparatus of claim 11 , wherein said bias (b ni ) related to at least one neuron is put at zero when its corresponding weights vector (w ni ) is put at zero.
18 . The apparatus of claim 11 , wherein said training algorithm is the Gradient Descent algorithm or the Stochastic Gradient Descent algorithm.
19 . The apparatus of claim 11 , comprising an output phase wherein processed data, such as pruned parameters {{w* ni ,b* ni }} and/or pruned model (M*) of a resulting pruned neural network, are provided as output.
20 . The apparatus of claim 19 , wherein said processed data are transmitted to a device.