PRUNING FILTERS FOR EFFICIENT CONVOLUTIONAL NEURAL NETWORKS FOR IMAGE RECOGNITION IN SURVEILLANCE APPLICATIONS
Systems and methods for pruning a convolutional neural network (CNN) for surveillance with image recognition are described, including extracting convolutional layers from a trained CNN, each convolutional layer including a kernel matrix having at least one filter formed in a corresponding output channel of the kernel matrix, and a feature map set having a feature map corresponding to each filter. An absolute kernel weight is determined for each kernel and summed across each filter to determine a magnitude of each filter. The magnitude of each filter is compared with a threshold and removed if it is below the threshold. A feature map corresponding to each of the removed filters is removed to prune the CNN of filters. The CNN is retrained to generate a pruned CNN having fewer convolutional layers to efficiently recognize and predict conditions in an environment being surveilled.
1 . A method for pruning a convolutional neural network (CNN) for surveillance with image recognition, the method comprising:
extracting at least one convolutional layer from a trained CNN, each convolutional layer including a kernel matrix having at least one filter formed in a corresponding output channel of the kernel matrix, and feature map set having a feature map corresponding to each of the at least one filter;
determining an absolute kernel weight for each kernel in the kernel matrix;
summing the absolute kernel weights of each kernel in each of the at least one filter to determine a magnitude of each filter;
comparing the magnitude of each filter with a threshold and removing one or more filters that are below the threshold;
removing a feature map corresponding to each of the removed filters to prune the CNN of filters; and
retraining the CNN upon pruning the removed filters to generate a pruned CNN having fewer convolutional layers to efficiently recognize and predict conditions in an environment being surveilled.
2 . The method as recited in claim 1 , further comprising removing the kernels of the removed filter from subsequent kernel matrices.
3 . The method as recited in claim 2 , further comprising, upon removing the kernels of the removed filter from subsequent kernel matrices, pruning the subsequent kernel matrices.
4 . The method as recited in claim 1 , further comprising iteratively retraining the CNN upon pruning each convolutional layer.
5 . The method as recited in claim 1 , further comprising retraining the CNN upon pruning every convolutional layer.
6 . The method as recited in claim 1 , wherein the threshold is a value corresponding to a minimum absolute kernel weight sum.
7 . The method as recited in claim 1 , wherein the threshold is a value corresponding to a number of filters having the smallest absolute kernel weight sums to be removed.
8 . A non-transitory computer readable storage medium comprising a computer readable program for surveillance with image recognition using a pruned convolutional neural network (CNN), wherein the computer readable program when executed on a computer causes the computer to perform the steps of:
extracting at least one convolutional layer from a trained CNN, each convolutional layer including a kernel matrix having at least one filter formed in a corresponding output channel of the kernel matrix, and feature map set having a feature map corresponding to each of the at least one filter;
determining an absolute kernel weight for each kernel in the kernel matrix;
summing the absolute kernel weights of each kernel in each of the at least one filter to determine a magnitude of each filter;
comparing the magnitude of each filter with a threshold and removing one or more filters that are below the threshold;
removing a feature map corresponding to each of the removed filters to prune the CNN of filters; and
retraining the CNN upon pruning the removed filters to generate a pruned CNN having fewer convolutional layers to efficiently recognize and predict conditions in an environment being surveilled.
9 . The computer readable program as recited in claim 8 , further comprising removing the kernels of the removed filter from subsequent kernel matrices.
10 . The computer readable program as recited in claim 9 , further comprising, upon removing the kernels of the removed filter from subsequent kernel matrices, pruning the subsequent kernel matrices.
11 . The computer readable program as recited in claim 8 , further comprising iteratively retraining the CNN upon pruning each convolutional layer.
12 . The computer readable program as recited in claim 8 , further comprising retraining the CNN upon pruning every convolutional layer.
13 . The computer readable program as recited in claim 8 , wherein the threshold is a value corresponding to a minimum absolute kernel weight sum.
14 . The computer readable program as recited in claim 8 , wherein the threshold is a value corresponding to a number of filters having the smallest absolute kernel weight sums to be removed.
15 . An image recognition system for surveillance, the system comprising:
an image capture device for capturing images of an environment to be surveilled;
an image recognition system in an embedded computing device included in the image capture device configured to perform image recognition with a pruned CNN, the image recognition system including:
an absolute kernel weight summer configured to determining an absolute kernel weight for each kernel in the kernel matrix and sum the absolute kernel weights of each kernel in a filter corresponding to each of at least one output channel of a kernel matrix to determine a magnitude of each filter, each filter corresponding to a feature map;
a threshold comparison unit configured to comparing the magnitude of each filter with a threshold and removing one or more filters that are below the threshold;
a layer updater configured to removing a feature map corresponding to each of the removed filters to prune the CNN of filters and generate the pruned CNN;
a long short-term memory network (LSTM) for predicting feature actions;
an action network for generating class probabilities of feature actions; and
a notification device for notifying a user of the class probabilities.
16 . The system as recited in claim 15 , wherein the layer update is further configured to remove the kernels of the removed filter from subsequent kernel matrices.
17 . The system as recited in claim 15 , wherein the image recognition system is further configured to iteratively retrain the CNN upon pruning each convolutional layer.
18 . The system as recited in claim 15 , wherein the image recognition system is further configured to retrain the CNN upon pruning every convolutional layer.
19 . The system as recited in claim 15 , wherein the threshold is a value corresponding to a minimum absolute kernel weight sum.
20 . The system as recited in claim 15 , wherein the threshold is a value corresponding to a number of filters having the smallest absolute kernel weight sums to be removed.