Activation based dynamic network pruning
A method for controlling operations of a machine learning model is performed by one or more processors and includes determining a threshold for skipping an operation, acquiring an activation value associated with at least one layer included in the machine learning model, determining whether the activation value is less than the threshold, and if the activation value is less than the threshold, controlling the operations of the machine learning model such that an operation associated with the activation value is skipped in the machine learning model.
1. A method for controlling operations of a machine learning model, the method being performed by one or more processors and comprising:
receive a first command to statistically determine a threshold for skipping an operation associated with a node and continuously maintain the threshold;
receive a second command to dynamically determine the threshold for skipping the operation associated with the node;
dynamically determining the threshold for skipping the operation associated with the node based on a distribution of a plurality of activation values associated with at least one layer included in the machine learning model, the machine learning model comprising a plurality of layers including an input layer that receives an input signal and an output layer that outputs an output signal and a plurality of hidden layers positioned between the input layer and the output layer to receive a signal from the input layer, extract features, and transmit the features to the output layer, wherein the plurality of layers includes a first layer including a first set of nodes and a second layer including a second set of nodes;
acquiring an activation value associated with at least one layer included in the machine learning model;
determining whether the activation value is less than the threshold; and
if the activation value is less than the threshold, controlling the operations of the machine learning model such that an operation associated with the activation value is skipped in the machine learning model,
wherein the activation value includes an output value from the first layer included in the machine learning model, and
the controlling the operations of the machine learning model includes, if operations associated with the second layer into which an output value from the first layer is input are performed and the output value from the first layer is less than the threshold, controlling the operations of the machine learning model by transmitting a skip command associated with the output value and storing the output value in a memory without changing the output value to zero such that an operation associated with the output value from the first layer is skipped in the second layer, performing the machine learning model by performing operations associated with the second set of nodes of the second layer and associated with values unassociated with the skip command and skipping the operations associated with the second set of nodes of the second layer and associated with the output value associated with the skip command regardless of whether the output value stored in the memory is zero;
wherein the determining the threshold includes:
when the activation value is less than the threshold, decreasing the threshold by a predetermined amount;
increasing a counter value indicating a time during which the threshold does not change when it is determined that the activation value is equal to or greater than the threshold; and
if the counter value reaches a predetermined value, resetting the counter value and updating the threshold such that the threshold is increased by the predetermined amount.
2. The method according to claim 1 , wherein the activation value is expressed as a floating-point number in a predetermined format,
wherein the method further comprising:
acquiring an exponent from the activation value expressed as the floating-point number;
determining whether the acquired exponent is less than a threshold; and
if the acquired exponent is less than the threshold, controlling the operations of the machine learning model such that an operation associated with the activation value is skipped in the machine learning model.
3. The method according to claim 2 , wherein the threshold is a value obtained by multiplying 2 by a predetermined number of times.
4. The method according to claim 2 , wherein each of the activation value and the threshold is a binary number.
5. The method according to claim 2 , further comprising, before the acquiring the activation value, determining the threshold based on a distribution of a plurality of activation values associated with the at least one layer.
6. The method according to claim 5 , wherein the determining the threshold includes:
determining a target activation value for comparison from a plurality of activation values associated with the at least one layer;
acquiring a target exponent from the determined target activation value; and
if the acquired target exponent is less than the threshold, updating the threshold such that the threshold is decreased by a predetermined amount.
7. The method according to claim 6 , wherein the updating the threshold includes decreasing the threshold by dividing the threshold by 2.
8. The method according to claim 1 , wherein the updating the threshold includes updating the threshold by multiplying the threshold by 2.
9. The method according to claim 2 , wherein the acquiring the activation value includes acquiring the activation value from a memory, wherein the activation value includes an output value from a first layer included in the machine learning model, and
the controlling the operations of the machine learning model includes,
if operations associated with a second layer into which the output value from the first layer is input are performed, and an exponent of the output value from the first layer is less than the threshold, controlling the operations of the machine learning model such that an operation associated with the activation value is skipped.
10. A non-transitory computer-readable recording medium storing instructions for execution by the one or more processors that, when executed by the one or more processors, cause the one or more processors to perform the method according to claim 1 .
11. A processing system comprising: a memory storing one or more instructions; and a processor configured to execute one or more instructions in the memory to:
receive a first command to statistically determine a threshold for skipping an operation associated with a node and continuously maintain the threshold;
receive a second command to dynamically determine the threshold for skipping the operation associated with the node;
dynamically determine the threshold for skipping the operation associated with the node based on a distribution of a plurality of activation values associated with at least one layer included in a machine learning model, the machine learning model comprising a plurality of layers including an input layer that receives an input signal and an output layer that outputs an output signal and a plurality of hidden layers positioned between the input layer and the output layer to receive a signal from the input layer, extract features, and transmit the features to the output layer, wherein the plurality of layers includes a first layer including a first set of nodes and a second layer including a second set of nodes;
acquire an activation value associated with at least one layer included in a machine learning model;
determine whether the activation value is less than the threshold; and
if the activation value is less than the threshold, control operations of the machine learning model such that an operation associated with the activation value is skipped in the machine learning model,
wherein the activation value includes an output value from the first layer included in the machine learning model, and
the processor is further configured to, if operations associated with the second layer into which an output value from the first layer is input are performed, and the output value from the first layer is less than the threshold, control the operations of the machine learning model by transmitting a skip command associated with the output value and storing the output value in a memory without changing the output value to zero such that an operation associated with the output value from the first layer is skipped in the second layer,
performing the machine learning model by performing operations associated with the second set of nodes of the second layer and associated with values unassociated with the skip command and skipping the operations associated with the second set of nodes of the second layer and associated with the output value associated with the skip command regardless of whether the output value stored in the memory is zero;
wherein the determining the threshold includes:
when the activation value is less than the threshold, decreasing the threshold by a predetermined amount;
increasing a counter value indicating a time during which the threshold does not change when it is determined that the activation value is equal to or greater than the threshold; and
if the counter value reaches a predetermined value, resetting the counter value and updating the threshold such that the threshold is increased by the predetermined amount.