Method and apparatus for compressing artificial neural network
A method and apparatus for compressing an artificial neural network may acquire weights corresponding to an artificial neural network trained in advance, wherein the artificial neural network includes a plurality of layers, and a processor configured to generate data for acquiring a change of behavior of the artificial neural network due to pruning of the artificial neural network based on the weights, determine a pruning threshold for pruning of the artificial neural network based on the change of the behavior of the artificial neural network, and compress the neural network based on the pruning threshold.
1 . A processor-implemented method of compressing a previously trained artificial neural network, the method comprising:
generating, based on weights corresponding to the artificial neural network of a plurality of layers, new image information for determining a change in behavior of the artificial neural network due to pruning, with the new image information being generated by iteratively modifying input image information until a class predicted by the artificial neural network based on the modified input image information is a target class from among a plurality of classes;
generating a corresponding pruning sensitivity for each of the plurality of layers in the artificial neural network using the new image information, which includes the modified input image information, each of the corresponding pruning sensitivities-indicating a degree to which the behavior of the artificial neural network changes when a respective layer is pruned;
determining, respectively based on the corresponding pruning sensitivities, a corresponding layer-specific pruning threshold for pruning each of the plurality of layers;
generating a compressed artificial neural network for the plurality of layers by pruning, for each of the plurality of layers, the artificial neural network based on the corresponding layer-specific pruning threshold; and
generating a recognition result or a classification result by executing the compressed neural network,
wherein the new image information is used, in the generating of the corresponding pruning sensitivity for each of the plurality of layers, to evaluate changes in the behavior of the artificial neural network when different layers are pruned, thereby eliminating a requirement for using training data for determining pruning thresholds,
wherein, as a result of the generating of the new image information, the new image information comprises a corresponding modified input image information for each of multiple classes of the plurality of classes, and
wherein the input image information is noise image information.
2 . The method of claim 1 , wherein the generating of the corresponding pruning sensitivity for each of the plurality of layers comprises:
generating, respectively for each of the plurality of layers, the corresponding pruning sensitivity by gradually increasing a percentile-based pruning threshold for a corresponding layer, in response to the behavior of the artificial neural network being maintained by a corresponding pruning of the corresponding layer.
3 . The method of claim 1 , wherein the corresponding pruning sensitivities are determined based on at least one of a distribution of weights corresponding to the plurality of layers or a form of connection between the plurality of layers.
4 . The method of claim 1 , wherein the determining of the corresponding pruning threshold for pruning each of the plurality of layers comprises:
selecting a layer from among the plurality of layers in an ascending order of the corresponding pruning sensitivities;
determining a percentile-based pruning threshold corresponding to the selected layer such that the behavior of the artificial neural network is maintained by a corresponding pruning of the selected layer; and
repeating the selecting of the layer and the determining of the percentile-based pruning threshold corresponding to the selected layer, until all remaining layers of the plurality of layers have been selected as the selected layer.
5 . The method of claim 1 , wherein the determining of the corresponding pruning threshold for pruning each of the plurality of layers comprises:
selecting a layer from among the plurality of layers in an ascending order of the pruning sensitivities;
pruning the selected layer to determine a percentile-based pruning threshold; and
determining a percentile-based pruning threshold, in response to upper k classes predicted by the artificial neural network before the pruning being included in upper p classes predicted by the artificial neural network after the pruning, wherein k and p each are a natural number and k≤p.
6 . The method of claim 5 , wherein the determining of the percentile-based pruning threshold further comprises:
increasing the percentile-based pruning threshold by a set interval, in response to the behavior of the artificial neural network being maintained by the pruning of the selected layer.
7 . The method of claim 1 , wherein the degree to which the behavior of the artificial neural network changes when the respective layer is pruned is measured based on whether outputs of the artificial neural network before and after the pruning of the respective layer satisfy a decision criterion.
8 . The method of claim 7 , wherein the decision criterion comprises a condition that upper p classes predicted by a pruned artificial neural network comprise upper k classes predicted by an unpruned artificial neural network, wherein k and p each are a natural number and k≤p.
9 . The method of claim 1 ,
wherein each of the corresponding layer-specific pruning thresholds comprises: a respective percentile-based pruning threshold, and
wherein the pruning of the artificial neural network comprises: applying, for each of a plurality of kernels in a corresponding layer of the plurality of layers, a magnitude-based pruning threshold to a corresponding kernel based on the respective percentile-based pruning threshold of the corresponding layer-specific pruning threshold of the corresponding layer.
10 . The method of claim 9 , wherein the pruning of the artificial neural network comprises removing a percentage of the weights of the artificial neural network, wherein the percentage of the weights corresponds to the respective percentile-based pruning thresholds.
11 . The method of claim 1 , wherein the input image information is a random noise image.
12 . The method of claim 1 , wherein the generating of the new image information comprises:
back-propagating a cross-entropy loss between a one-hot vector corresponding to the target class and the class predicted by the artificial neural network.
13 . The method of claim 12 , wherein the weights are fixed without changing during backpropagation of the cross-entropy loss.
14 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 .
15 . An apparatus for compressing a previously trained artificial neural network, the apparatus comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
generate, based on weights corresponding to the artificial neural network of a plurality layers, new image information for determining a change in behavior of the artificial neural network due to pruning, with the new image information being generated by iteratively modifying input image information until a class predicted by the artificial neural network based on the modified input image information is a target class from among a plurality of classes;
generate a corresponding pruning sensitivity for each of the plurality of layers in the artificial neural network using the new image information, which includes the modified input image information, each of the corresponding pruning sensitivities indicating a degree to which the behavior of the artificial neural network changes when a respective layer is pruned;
determine, respectively based on the corresponding pruning sensitivities, a corresponding layer-specific pruning threshold for pruning each of the plurality of layers;
generate a compressed artificial neural network for the plurality of layers by pruning, for each of the plurality of layers, the artificial neural network based on the corresponding layer-specific pruning threshold; and
generate a recognition result or a classification result by executing the compressed neural network,
wherein the new image information is used, in the generation of the corresponding pruning sensitivity for each of the plurality of layers, to evaluate changes in the behavior of the artificial neural network when different layers are pruned, thereby eliminates a requirement for using training data for determining pruning thresholds,
wherein, as a result of the generation of the new image information, the new image information comprises a corresponding modified input image information for each of multiple classes of the plurality of classes, and
wherein the input image information is noise image information.
16 . The apparatus of claim 15 , wherein the generation of the corresponding pruning sensitivity for each of the plurality of layers comprises a generation, respectively for each of the plurality of layers, the corresponding pruning sensitivity by gradually increasing a percentile-based pruning threshold for a corresponding layer, in response to the behavior of the artificial neural network being maintained by a corresponding pruning of the corresponding layer.
17 . The apparatus of claim 15 , wherein, for the determination of the corresponding pruning threshold for pruning each of the plurality of layers, the execution of the instructions causes the apparatus to determine the corresponding pruning sensitivities based on at least one of a distribution of weights corresponding to the plurality of layers or a form of connection between the plurality of layers.
18 . The apparatus of claim 15 , wherein, for the determination of the corresponding pruning threshold for pruning each of the plurality of layers, the execution of the instructions causes the apparatus to:
select a layer from among the plurality of layers in an ascending order of the corresponding pruning sensitivities;
determine a percentile-based pruning threshold corresponding to the selected layer such that the behavior of the artificial neural network is maintained by a corresponding pruning of the selected layer' and
repeat the selection of the layer and the determination of the percentile-based pruning threshold corresponding to the selected layer, until all remaining layers of the plurality of layers have been selected as the selected layer.
19 . The apparatus of claim 15 , wherein the degree to which the behavior of the artificial neural network changes when the respective layer is pruned is measured based on whether outputs of the artificial neural network before and after the pruning of the respective layer satisfy a decision criterion.
20 . The apparatus of claim 19 , wherein the decision criterion comprises a condition that upper p classes predicted by a pruned artificial neural network comprise upper k classes predicted by an unpruned artificial neural network, wherein k and p each are a natural number and k≤p.
21 . The apparatus of claim 15 ,
wherein each of the corresponding layer-specific pruning thresholds comprises: a respective percentile-based pruning threshold, and
wherein, for the pruning of the artificial neural network, the execution of the instructions cause the apparatus to apply, for each of a plurality of kernels in a corresponding layer of the plurality of layers, a magnitude-based pruning threshold to a corresponding kernel based on the respective percentile-based pruning threshold of the corresponding layer-specific pruning threshold of the corresponding layer.
22 . The apparatus of claim 15 , wherein the input image information is a random noise image.
23 . The apparatus of claim 15 , wherein, for the generation of the new image information, the execution of the instructions causes the apparatus back-propagate a cross-entropy loss between a one-hot vector corresponding to the target class and the class predicted by the artificial neural network.
24 . The apparatus of claim 23 , wherein the weights are fixed without changing during backpropagation of the cross-entropy loss.
25 . The apparatus of claim 15 , wherein the apparatus is at least one of an advanced driver assistance system (ADAS), a head-up display (HUD) device, a three-dimensional (3D) digital information display (DID), a navigation device, a neuromorphic device, a 3D mobile device, a smartphone, a smart television (TV), a smart vehicle, an Internet of Things (IoT) device, a medical device, or a measurement device.
26 . The method of claim 1 , wherein the generating of the new image information comprises performing the iterative modifying of the input image information for each class, as respectively different target classes in each different iterative modifying of the input image information, of the plurality of classes.
27 . The method of claim 1 , wherein the new image information comprises the corresponding modified input image information for each class of the plurality of classes.