Neural network pruning method and system via layerwise analysis
Embodiments disclosed herein allowed neural networks to be pruned. The inputs and outputs generated by a reference neural network are used to prune the reference neural network. The pruned neural network may have a subset of the weights that are in the reference neural network.
1. A method, comprising:
obtaining a first neural network, wherein:
the first neural network is trained using a first set of training data;
the first neural network comprises a first set of nodes; and
the first neural network comprises a first set of connections that interconnect the first set of nodes;
generating a second neural network based on the first neural network, wherein generating the second neural network comprises
analyzing a set of intermediate layers of the first neural network,
determining subsets of weights for each intermediate layer of the intermediate layers, including generating, for each intermediate layer, a respective filter based on an input provided to the intermediate layer and a reference output, wherein the respective filter comprises a respective one of the subsets of weights, and
generating the second neural network based on the subsets of weights for each intermediate layer, wherein:
the second neural network comprises a second set of connections interconnecting a second set of nodes and the second set of connections comprises a subset of the first set of connections; and
the second neural network is generated without using the first set of training data.
2. The method of claim 1 , wherein the input is generated by:
stacking a first set of input feature maps to generate a first combined feature map, wherein the input comprises the first combined feature map.
3. The method of claim 2 , wherein the first set of input feature maps are generated by a first filter of the first neural network.
4. The method of claim 1 , wherein the reference output is generated by:
flattening an output feature map to generate a vector, wherein the reference output comprises the vector.
5. The method of claim 4 , wherein the output feature map is generated based on a second filter of the first neural network.
6. The method of claim 1 , wherein the subsets of weights for each intermediate layer are determined simultaneously for each intermediate layer.
7. The method of claim 1 , wherein generating the second neural network comprises:
generating the second neural network without training the second neural network.
8. The method of claim 1 , wherein the second set of nodes comprises a subset of the first set of nodes.
9. A method, comprising:
obtaining a first neural network, wherein:
the first neural network is trained by passing a set of training data through the first neural network a first number of times;
the first neural network comprises a first set of nodes; and
the first neural network comprises a first set of connections that interconnect the first set of nodes;
generating a second neural network based on the first neural network, wherein generating the second neural network comprises
analyzing a set of intermediate layers of the first neural network,
determining subsets of weights for each intermediate layer of the intermediate layers, including generating, for each intermediate layer, a respective filter based on an input provided to the intermediate layer and a reference output, wherein the respective filter comprises a respective one of the subsets of weights, and
generating the second neural network based on the subsets of weights for each intermediate layer, wherein:
the second neural network comprises a second set of connections interconnecting a second set of nodes and the second set of connections comprises a subset of the first set of connections; and
the second neural network is generated by passing the set of training data through the second neural network a second number of times, wherein the second number of times is smaller than the first number of times.
10. The method of claim 9 , wherein the input is generated by:
stacking a first set of input feature maps to generate a first combined feature map, wherein the input comprises the first combined feature map.
11. The method of claim 10 , wherein the first set of input feature maps are generated by a previous layer of the second neural network.
12. The method of claim 9 , wherein the reference output is generated by:
flattening an output feature map to generate a vector, wherein the reference output comprises the vector.
13. The method of claim 12 , wherein the output feature map is generated based on a second filter of the first neural network.
14. The method of claim 9 , wherein the subsets of weights for each intermediate layer are determine sequentially, layer by layer.
15. The method of claim 9 , wherein the second set of nodes comprises a subset of the first set of nodes.
16. An apparatus, comprising:
a memory configured to store data;
a processor coupled to the memory, the processor configured to:
obtain a first neural network, wherein:
the first neural network is trained using a first set of training data;
the first neural network comprises a first set of nodes; and
the first neural network comprises a first set of connections that interconnect the first set of nodes;
generate a second neural network based on the first neural network, wherein generating the second neural network comprises
analyzing a set of intermediate layers of the first neural network,
determining subsets of weights for each intermediate layer of the intermediate layers, including generating, for each intermediate layer, a respective filter based on an input provided to the intermediate layer and a reference output, wherein the respective filter comprises a respective one of the subsets of weights, and
generating the second neural network based on the subsets of weights for each intermediate layer, wherein:
the second neural network comprises a second set of connections interconnecting a second set of nodes and the second set of connections comprises a subset of the first set of connections; and
the second neural network is generated without using the first set of training data.
17. The apparatus of claim 16 , wherein the input is generated by:
stacking a first set of input feature maps to generate a first combined feature map, wherein the input comprises the first combined feature map.