FILTER BASED PRUNING TECHNIQUES FOR CONVOLUTIONAL NEURAL NETWORKS
A system and method for filter based pruning of a convolutional neural network (CNN), is disclosed. The method includes initializing a CNN, the CNN including a plurality of filters, each filter associated with a weight and a filter factor; providing the CNN with a training input; adjusting a weight of a filter of the plurality of filters in response to processing the training input; adjusting a filter factor of the filter of the plurality of filters in response to processing the training input; pruning the CNN by removing the filter in response to detecting that a value of the filter factor is below a predefined threshold after training is complete; storing a trained pruned CNN based on the initialized CNN; and processing an input with the trained CNN.
1 . A method for filter based pruning of a convolutional neural network (CNN), comprising:
initializing a CNN, the CNN including a plurality of filters, each filter associated with a weight and a filter factor;
providing the CNN with a training input;
adjusting a weight of a filter of the plurality of filters in response to processing the training input;
adjusting a filter factor of the filter of the plurality of filters in response to processing the training input;
pruning the CNN by removing the filter in response to detecting that a value of the filter factor is below a predefined threshold after training is complete;
storing a trained pruned CNN based on the initialized CNN; and
processing an input with the trained CNN.
2 . The method of claim 1 , further comprising:
determining a number of single instruction multiple data (SIMD) processing units;
removing a number of filters based on the filter factor, such that a second number of remaining filters is a whole multiple of the number of SIMD processing units.
3 . The method of claim 2 , further comprising:
selecting a number of second filters, each having a filter factor value which exceeds the predefined threshold; and
removing a number of filters, the number of filters equal to a number of filters having a filter factor value below the predefined threshold added to the number of second filters.
4 . The method of claim 1 , further comprising:
determining a number of single instruction multiple data (SIMD) processing units;
removing a number of filters based on the filter factor, such that the number of removed filters is a whole multiple of the number of SIMD processing units.
5 . The method of claim 4 , further comprising:
selecting a number of second filters, each having a filter factor value which exceeds the predefined threshold; and
removing a number of filters, the number of filters equal to a number of filters having a filter factor value below the predefined threshold added to the number of second filters.
6 . The method of claim 1 , wherein the filter factor of each filter of the plurality of filters includes a value selected between a lower limit value and an upper limit value.
7 . The method of claim 1 , wherein a weight value, a filter factor value, and a combination thereof is stored as any one of: a fixed point value, a floating point value, an integer value, and any combination thereof.
8 . The method of claim 1 , wherein the trained pruned CNN includes only filters having a filter factor above a predefined threshold.
9 . The method of claim 1 , further comprising:
applying a pruning technique only on a predetermined number of layers of the CNN.
10 . The method of claim 1 , further comprising:
applying a hyperparameter value in training the CNN.
11 . The method of claim 1 , wherein a loss function of the CNN includes a base loss function and a filter based pruning loss function.
12 . A non-transitory computer-readable medium storing a set of instructions for filter based pruning of a convolutional neural network (CNN), the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
initialize a CNN, the CNN including a plurality of filters, each filter associated with a weight and a filter factor;
provide the CNN with a training input;
adjust a weight of a filter of the plurality of filters in response to processing the training input;
adjust a filter factor of the filter of the plurality of filters in response to processing the training input;
prune the CNN by removing the filter in response to detecting that a value of the filter factor is below a predefined threshold after training is complete;
store a trained pruned CNN based on the initialized CNN; and
process an input with the trained CNN.
13 . A system for filter based pruning of a convolutional neural network (CNN) comprising:
a processing circuitry; and
a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:
initialize a CNN, the CNN including a plurality of filters, each filter associated with a weight and a filter factor;
provide the CNN with a training input;
adjust a weight of a filter of the plurality of filters in response to processing the training input;
adjust a filter factor of the filter of the plurality of filters in response to processing the training input;
prune the CNN by removing the filter in response to detecting that a value of the filter factor is below a predefined threshold after training is complete;
store a trained pruned CNN based on the initialized CNN; and
process an input with the trained CNN.
14 . The system of claim 13 , wherein the memory contains further instructions which when executed by the processing circuitry further configure the system to:
determine a number of single instruction multiple data (SIMD) processing units; and
remove a number of filters based on the filter factor, such that a second number of remaining filters is a whole multiple of the number of SIMD processing units.
15 . The system of claim 14 , wherein the memory contains further instructions which when executed by the processing circuitry further configure the system to:
select a number of second filters, each having a filter factor value which exceeds the predefined threshold; and
remove a number of filters, the number of filters equal to a number of filters having a filter factor value below the predefined threshold added to the number of second filters.
16 . The system of claim 13 , wherein the memory contains further instructions which when executed by the processing circuitry further configure the system to:
determine a number of single instruction multiple data (SIMD) processing units; and
remove a number of filters based on the filter factor, such that the number of removed filters is a whole multiple of the number of SIMD processing units.
17 . The system of claim 16 , wherein the memory contains further instructions which when executed by the processing circuitry further configure the system to:
select a number of second filters, each having a filter factor value which exceeds the predefined threshold; and
remove a number of filters, the number of filters equal to a number of filters having a filter factor value below the predefined threshold added to the number of second filters.
18 . The system of claim 13 , wherein the filter factor of each filter of the plurality of filters includes a value selected between a lower limit value and an upper limit value.
19 . The system of claim 13 , wherein a weight value, a filter factor value, and a combination thereof is stored as any one of: a fixed point value, a floating point value, an integer value, and any combination thereof.
20 . The system of claim 13 , wherein the trained pruned CNN includes only filters having a filter factor above a predefined threshold.
21 . The system of claim 13 , wherein the memory contains further instructions which when executed by the processing circuitry further configure the system to:
apply a pruning technique only on a predetermined number of layers of the CNN.
22 . The system of claim 13 , wherein the memory contains further instructions which when executed by the processing circuitry further configure the system to:
apply a hyperparameter value in training the CNN.
23 . The system of claim 13 , wherein a loss function of the CNN includes a base loss function and a filter based pruning loss function.