Dynamic neural network surgery
Techniques related to compressing a pre-trained dense deep neural network to a sparsely connected deep neural network for efficient implementation are discussed. Such techniques may include iteratively pruning and splicing available connections between adjacent layers of the deep neural network and updating weights corresponding to both currently disconnected and currently connected connections between the adjacent layers.
1. A method for compressing a deep neural network, comprising:
receiving weights of the deep neural network, the deep neural network having layers that comprise an input layer, one or more hidden layers, and an output layer, the weights corresponding to available connections between the layers;
generating a sparsely connected deep neural network by disconnecting a connection of the available connections based on a value of a weight corresponding to the connection, wherein the connection is between a first node of a first layer and a second node of a second layer, and both the first node and the second node are maintained in the sparsely connected deep neural network; and
generating another sparsely connected deep neural network by:
after disconnecting the connection, updating the value of the weight corresponding to the connection between the first node and the second node by further training the sparsely connected deep neural network,
determining whether the updated value of the weight is above a threshold, and
in response to determining that the updated value of the weight is above the threshold, reconnecting the connection between the first node and the second node,
wherein the another sparsely connected deep neural network, comprises the connection between the first node and the second node and has less available connections than the deep neural network.
2. The method of claim 1 , further comprising:
generating a connection matrix comprising a plurality of indicators each indicating whether a corresponding available connection between two layers is connected or not connected, wherein generating the connection matrix comprises applying a discriminative function onto the weights of the deep neural network to determine each indicator of the connection matrix; and
updating the connection matrix after the value of the weight is updated.
3. The method of claim 2 , wherein applying the discriminative function comprises:
comparing an individual weight to the threshold; and
including a disconnect indicator in the connection matrix after a determination that the individual weight is below the threshold.
4. The method of claim 2 , wherein applying the discriminative function comprises:
comparing an individual weight to a first threshold and to a second threshold greater than first threshold; and
including a disconnect indicator in the connection matrix after a determination that the individual weight is below the first threshold, or including a connect indicator in the connection matrix after a determination that the individual weight is above the second threshold.
5. The method of claim 1 , wherein updating the value of the weight comprises:
inputting a training data set into the sparsely connected deep neural network;
determining a network loss based on the training data set and the weights corresponding to the available connections; and
updating the value of the weight based on the network loss.
6. The method of claim 5 , wherein the training data set is a first training data set, the method further comprises:
before generating the sparsely connected deep neural network, training the deep neural network based on a second training data set to determine values of the weights corresponding to the available connections, wherein the first training data set is a subset of the second training data set.
7. The method of claim 1 , wherein the sparsely connected deep neural network and the another sparsely connected deep neural network are generated in an iteration of an iterative compressing process, and the iterative compressing process comprises:
stochastically determining, for the iteration, an activation indicator indicating whether to compress the deep neural network in the iteration; and
generating the sparsely connected deep neural network and the another sparsely connected deep neural network when activation indicator indicates to compress the deep neural network in the iteration.
8. The method of claim 7 , wherein stochastically determining the activation indicator comprises applying a probability function based on an iteration number of the iteration, and the probability function is a monotonically non-increasing probability function.
9. The method of claim 1 , wherein the deep neural network comprises a fully connected neural network, a convolutional neural network, or a recurrent neural network.
10. The method of claim 1 , wherein the deep neural network comprises a convolutional neural network, and the weights comprise coefficients of a kernel of the convolutional layer.
11. The method of claim 1 , wherein the sparsely connected deep neural network is
to receive input data for classification and to generate classification data.
12. The method of claim 1 , wherein the first node or the second node has at least one other connection to another node.
13. A system for compressing a deep neural network comprising:
memory to store weights of the deep neural network, the deep neural network having layers that comprise an input layer, one or more hidden layers, and an output layer, the weights corresponding to available connections between the layers; and
a processor or processor circuitry coupled to the memory, the processor or processor circuitry to:
generate a sparsely connected deep neural network by disconnecting a connection of the available connections based on a value of a weight corresponding to the connection, wherein the connection is between a first node of a first layer and a second node of a second layer, and both the first node and the second node are maintained in the sparsely connected deep neural network, and
generate another sparsely connected deep neural network by:
after disconnecting the connection, updating the value of the weight corresponding to the connection between the first node and the second node by further training the sparsely connected deep neural network,
determining whether the updated value of the weight is above a threshold, and
in response to determining that the updated value of the weight is above the threshold, reconnecting the connection between the first node and the second node,
wherein the another sparsely connected deep neural network comprises the connection between the first node and the second node and has less available connections than the deep neural network.
14. The system of claim 13 , wherein the processor or processor circuitry is further to:
generate a connection matrix comprising a plurality of indicators each indicating whether a corresponding available connection between two layers is connected or not connected, wherein generating the connection matrix comprises applying a discriminative function onto the weights of the deep neural network to determine each indicator of the connection matrix; and
update the connection matrix after the value of the weight is updated.
15. The system of claim 14 , wherein to apply the discriminative function comprises the processor or processor circuitry to:
compare an individual weight to a first threshold and to a second threshold greater than first threshold; and
include a disconnect indicator in the connection matrix after a determination that the individual weight is below the first threshold, or include a connect indicator in the connection matrix after a determination that the individual weight is above the second threshold.
16. The system of claim 13 , wherein updating the value of the weight comprises:
inputting a training data set into the sparsely connected deep neural network;
determining a network loss based on the training data set and the weights corresponding to the available connections; and
updating the value of the weight based on the network loss.
17. The system of claim 13 , wherein the sparsely connected deep neural network and the another sparsely connected deep neural network are generated in an iteration of an iterative compressing process, and the iterative compressing process comprises:
stochastically determining, for the iteration, an activation indicator indicating whether to compress the deep neural network in the iteration; and
generating the sparsely connected deep neural network and the another sparsely connected deep neural network when the activation indicator indicates to compress the deep neural network in the iteration.
18. The system of claim 17 , wherein to stochastically determine the activation indicator comprises the processor or processor circuitry to apply a probability function based on an iteration number of the iteration, and the probability function is a monotonically non-increasing probability function.
19. The system of claim 13 , wherein the deep neural network comprises a convolutional neural network, and the weights comprise coefficients of a kernel of the convolutional layer.
20. At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to compress a deep neural network by:
receiving weights of the deep neural network, the deep neural network having layers that comprise an input layer, one or more hidden layers, and an output layer, the weights corresponding to available connections between the layers;
generating a sparsely connected deep neural network by disconnecting a connection of the available connections based on a value of a weight corresponding to the connection, wherein the connection is between a first node of a first layer and a second node of a second layer, and both the first node and the second node are maintained in the sparsely connected deep neural network; and
generating another sparsely connected deep neural network by:
after disconnecting the connection, updating the value of the weight corresponding to the connection between the first node and the second node by further training the sparsely connected deep neural network,
determining whether the updated value of the weight is above a threshold, and
in response to determining that the updated value of the weight is above the threshold, reconnecting the connection between the first node and the second node,
wherein the another sparsely connected deep neural network, comprises the connection between the first node and the second node and has less available connections than the deep neural network.
21. The non-transitory machine readable medium of claim 20 , wherein compressing the deep neural network further comprises:
generating a connection matrix comprising a plurality of indicators each indicating whether a corresponding available connection between two layers is connected or not connected, wherein generating the connection matrix comprises applying a discriminative function onto the weights of the deep neural network to determine each indicator of the connection matrix; and
updating the connection matrix after the value of the weight is updated.
22. The non-transitory machine readable medium of claim 21 , wherein applying the discriminative function comprises:
comparing an individual weight to the threshold; and
including a disconnect indicator in the connection matrix after a determination that the individual weight is below the threshold.
23. The non-transitory machine readable medium of claim 20 , wherein updating the value of the weight comprises:
inputting a training data set into the sparsely connected deep neural network;
determining a network loss based on the training data set and the weights corresponding to the available connections; and
updating the value of the weight based on the network loss.
24. The non-transitory machine readable medium of claim 20 , wherein the sparsely connected deep neural network and the another sparsely connected deep neural network are generated in an iteration of an iterative compressing process, and the iterative compressing process comprises:
stochastically determining, for the iteration, an activation indicator indicating whether to compress the deep neural network in the iteration; and
generating the sparsely connected deep neural network and the another sparsely connected deep neural network when the activation indicator indicates to compress the deep neural network in the iteration.
25. The non-transitory machine readable medium of claim 24 , wherein stochastically determining the activation indicator comprises applying a probability function based on an iteration number of the iteration, and the probability function is a monotonically non-increasing probability function.