Automatic filter pruning technique for convolutional neural networks
An automated pruning technique is proposed for reducing the size of a convolutional neural network. A large-sized network is trained and then connections between layers are explored to remove redundant parameters. Specifically, a scaling neural subnetwork is connected to the neural network and designed to infer importance of the filters in the neural network during training of the neural network. Output from the scaling neural subnetwork can then be used to remove filters from the neural network, thereby reducing the size of the convolutional neural network.
1. A computer-implemented method for pruning a convolutional neural network, the convolutional neural network includes a plurality of convolutional layers and each convolutional layer includes a plurality of neurons implemented in hardware, comprising:
for at least one convolutional layer in the convolutional neural network, connecting a scaling neural subnetwork to the at least one convolutional layer, where an input layer of the scaling neural subnetwork is configured to receive weights of the filters in the at least one convolutional layer and an output layer of the scaling neural network outputs a scale vector, where elements of the scale vector quantify importance of filters in the at least one convolutional layer;
training the convolutional neural network, including the scaling neural subnetworks connected thereto; and
for the at least one convolutional layer in the convolutional neural network, removing filters from the convolutional layer based on elements of a scale vector output by the respective scaling neural network, where filters are removed from the convolutional layer after training of the convolutional neural network.
2. The method of claim 1 wherein the input layer of the scaling neural subnetwork evaluates size of the filters in the at least one convolutional layer by applying a norm.
3. The method of claim 2 wherein the input layer of the scaling neural subnetwork evaluates similarity amongst filters in the at least one convolutional layer.
4. The method of claim 3 wherein the input layer of the scaling neural subnetwork evaluates similarity amongst filters in the at least one convolutional layer by computing a cross-correlation between each pair of filters in the at least one convolutional layer.
5. The method of claim 4 wherein an output layer of the scaling neural subnetwork is an activation function.
6. The method of claim 1 wherein connecting a scaling neural subnetwork to the at least one convolutional layer further comprises combining the scale vector from the scaling neural network with output from the respective convolutional layer.
7. The method of claim 1 further comprises training the convolutional neural network using a backpropagation method.
8. The method of claim 1 wherein removing filters from the at least one convolutional layer further includes comparing elements of the scale vector to a threshold and removing filters corresponding to the elements whose value is less than the threshold.
9. A computer-implemented method for pruning a convolutional neural network, the convolutional neural network includes a plurality of convolutional layers and each convolutional layer includes a plurality of neurons implemented in hardware, comprising:
configuring a scaling neural subnetwork to extract descriptors from filters of a convolutional layer of the convolutional neural network and infer importance of the filters;
for at least one convolutional layer in the convolutional neural network, connecting the scaling neural subnetwork to the respective convolutional layer, where an input layer of the scaling neural subnetwork is configured to receive weights of the filters in the respective convolutional layer and an output layer of the scaling neural subnetwork outputs a scale vector, such that elements of the scale vector quantify importance of filters in the respective convolutional layer;
training the convolutional neural network, including the scaling neural subnetworks connected thereto; and
removing filters from the at least one convolutional layer based on elements of a scale vector output by the respective scaling neural subnetwork, where filters are removed from the at least one convolutional layer after training of the convolutional neural network.
10. The method of claim 9 wherein the scaling neural subnetwork extracts descriptors from filters of a convolutional layer by applying a norm to weights of the filters.
11. The method of claim 9 wherein the scaling neural subnetwork extracts descriptors from filters of a convolutional layer by determining similarity amongst filters in the respective convolutional layer.
12. The method of claim 11 wherein the scaling neural subnetwork determines similarity amongst filters in the respective convolutional layer by computing a cross-correlation between each pair of filters in the respective convolutional layer.
13. The method of claim 9 wherein an output layer of the scaling neural subnetwork is an activation function.
14. The method of claim 9 wherein connecting a scaling neural subnetwork to the respective convolutional layer further comprises combining the scale vector from the scaling neural network with output from the respective convolutional layer.
15. The method of claim 9 further comprises training the convolutional neural network using a backpropagation method.
16. The method of claim 9 wherein removing filters from the convolutional layer further includes comparing elements of the scale vector to a threshold and removing filters corresponding to the elements whose value is less than the threshold.