Passive pruning of filters in a convolutional neural network
Methods and systems of training a neural network includes training a neural network based on training data. Weights of a layer of the neural network are multiplied by an attrition factor. A block of weights is pruned from the layer if the block of weights in the layer has a contribution to an output of the layer that is below a threshold.
1. A method of training a neural network, comprising:
training a neural network based on training data;
multiplying weights of a layer of the neural network by an attrition factor; and
pruning a block of weights from the layer if the block of weights in the layer has a contribution to an output of the layer that is below a threshold.
2. The method of claim 1 , wherein the attrition factor is a number less than one.
3. The method of claim 1 , wherein the contribution of a block of weights to the output of the layer is calculated as a percentage of a sum of absolute weights of the weights in the layer made up by a sum of absolute weights of the weights in the block of weights.
4. The method of claim 1 , further comprising pruning a filter in a subsequent layer in the neural network that corresponds to the pruned block of weights.
5. The method of claim 4 , further comprising pruning a block of weights in a subsequent layer in the neural network that corresponds to the pruned filter.
6. The method of claim 1 , wherein the neural network is a convolutional neural network.
7. The method of claim 1 , wherein training, multiplying, and pruning are repeated until output of the neural network is within a threshold difference from an expected output for a validation data set.
8. The method of claim 1 , wherein training the neural network comprises a forward pass using the training data, a backward pass, and a learning pass that updates weights of the neural network.
9. The method of claim 1 , wherein pruning a block of weights comprises removing a column or row of an array of weights.
10. A method of training a neural network, comprising:
training a convolutional neural network based on training data;
multiplying weights of a layer of the neural network by a number less than one; and
pruning a block of weights from the layer, pruning a filter corresponding to the block of weights in a subsequent layer in the neural network, and pruning a block of weights that corresponds to the pruned filter in a subsequent layer in the neural network, if the block of weights in the layer has a contribution to an output of the layer that is below a threshold, wherein the contribution of a block of weights to the output of the layer is calculated as a percentage of a sum of absolute weights of the weights in the layer made up by a sum of absolute weights of the weights in the block of weights.
11. A system for training a neural network, comprising:
a neural network;
a training module configured to train the neural network based on training data; and
a pruning module configured to multiply weights of a layer of the neural network by an attrition factor and to prune a block of weights from the layer if the block of weights in the layer has a contribution to an output of the layer that is below a threshold.
12. The system of claim 11 , wherein the attrition factor is a number less than one.
13. The system of claim 11 , wherein the pruning module is further configured to calculate the contribution of a block of weights to the output of the layer as a percentage of a sum of absolute weights of the weights in the layer made up by a sum of absolute weights of the weights in the block of weights.
14. The system of claim 11 , further wherein the pruning module is further configured to prune a filter in a subsequent layer in the neural network that corresponds to the pruned block of weights.
15. The system of claim 14 , wherein the pruning module is further configured to prune a block of weights in a subsequent layer in the neural network that corresponds to the pruned filter.
16. The system of claim 11 , wherein the neural network is a convolutional neural network.
17. The system of claim 11 , wherein the training module and the pruning module are further configured to repeat training, multiplying, and pruning until output of the neural network is within a threshold difference from an expected output for a validation data set.
18. The system of claim 11 , wherein the training module is further configured to train the neural network using a forward pass using the training data, a backward pass, and a learning pass that updates weights of the neural network.
19. The system of claim 11 , wherein the pruning module is further configured to remove a column or row of an array of weights.