Quantization-aware training with numerical overflow avoidance for neural networks
An apparatus and method for efficiently creating less computationally intensive nodes for a neural network. In various implementations, a computing system includes a memory that stores multiple input data values for training a neural network, and a processor. Rather than determine a bit width P of an integer accumulator of a node of the neural network based on bit widths of the input data values and corresponding weight values, the processor selects the bit width P during training. The processor adjusts the magnitudes of the weight values during iterative stages of training the node such that an L1 norm value of the weight values of the node does not exceed a corresponding weight magnitude limit.
1 . An apparatus comprising:
circuitry configured to:
select a first weight value with a first magnitude, based at least in part on an input data value used for training a node of a neural network;
replace the first weight value with a second weight value that has a second magnitude different from the first magnitude, based at least in part on the first magnitude and one or more magnitudes of a first set of weight values comprising one or more weight values generated previously for the node; and
generate an output value for the node based at least in part on the second magnitude.
2 . The apparatus as recited in claim 1 , wherein the circuitry is further configured to replace the first set of weight values with a second set of weight values that have one or more magnitudes less than a corresponding one of the one or more magnitudes of the first set of weight values.
3 . The apparatus as recited in claim 2 , wherein the second magnitude is less than the first magnitude.
4 . The apparatus as recited in claim 1 , wherein the circuitry is further configured to:
receive a bit width P of an accumulator value used in the node; and
generate a weight magnitude limit as two to a power of P less N, wherein:
N is a bit width of input data values of the node; and
each of P and N is a positive, non-zero integer.
5 . The apparatus as recited in claim 4 , wherein the circuitry is further configured to replace the first magnitude with the second magnitude, in response to determining a first sum of absolute values of the first magnitude and the one or more magnitudes of the first set of weight values is greater than the weight magnitude limit.
6 . The apparatus as recited in claim 5 , wherein the circuitry is further configured to:
generate a difference between the first sum and the weight magnitude limit;
generate an average value by dividing the difference by a second sum of a number of weight values in the first set of weight values and one for the first weight value;
generate an adjustment value based at least in part on the average value; and
generate the second magnitude as an absolute value of the first magnitude less the adjustment value.
7 . The apparatus as recited in claim 5 , wherein the circuitry is further configured to:
generate a ratio by dividing the weight magnitude limit by the first sum;
generate a scaling factor based at least in part on the ratio; and
generate the second magnitude is a multiplicative product of an absolute value of the first magnitude and the scaling factor.
8 . A method, comprising:
storing, by circuitry of a memory, a plurality of input data values for a neural network;
selecting, by circuitry of a processor, a first weight value with a first magnitude, based at least in part on an input data value used for training a node of a neural network; and
replacing, by the circuitry, the first weight value with a second weight value that has a second magnitude different from the first magnitude, based at least in part on the first magnitude and one or more magnitudes of a first set of weight values comprising one or more weight values generated previously for the node; and
generating, by the circuitry, an output value for the node based at least in part on the second magnitude.
9 . The method as recited in claim 8 , further comprising replacing, by the circuitry, the first set of weight values with a second set of weight values that have one or more magnitudes less than a corresponding one of the one or more magnitudes of the first set of weight values.
10 . The method as recited in claim 9 , wherein the second magnitude is less than the first magnitude.
11 . The method as recited in claim 8 , further comprising:
receiving, by the circuitry, a bit width P of an accumulator value used in the node; and
generating, by the circuitry, a weight magnitude limit as two to a power of P less N, wherein:
N is a bit width of input data values of the node; and
each of P and N is a positive, non-zero integer.
12 . The method as recited in claim 11 , further comprising replacing, by the circuitry, the first magnitude with the second magnitude, in response to determining a first sum of absolute values of the first magnitude and the one or more magnitudes of the first set of weight values is greater than the weight magnitude limit.
13 . The method as recited in claim 12 , further comprising:
generating, by the circuitry, a difference between the first sum and the weight magnitude limit;
generating, by the circuitry, an average value by dividing the difference by a second sum of a number of weight values in the first set of weight values and one for the first weight value;
generating, by the circuitry, an adjustment value based at least in part on the average value; and
generating, by the circuitry, the second magnitude as an absolute value of the first magnitude less the adjustment value.
14 . The method as recited in claim 13 , further comprising:
generating, by the circuitry, a ratio by dividing the weight magnitude limit by the first sum;
generating, by the circuitry, a scaling factor based at least in part on the ratio; and
generating, by the circuitry, the second magnitude is a multiplicative product of an absolute value of the first magnitude and the scaling factor.
15 . A computing system comprising:
a memory comprising circuitry configured to store a plurality of input data values for training a neural network;
a processor comprising circuitry configured to:
select a first weight value with a first magnitude, based at least in part on an input data value of the plurality of input data values used for training a node of a neural network;
replace the first weight value with a second weight value that has a second magnitude different from the first magnitude, based at least in part on the first magnitude and one or more magnitudes of a first set of weight values comprising one or more weight values generated previously for the node; and
generate an output value for the node based at least in part on the second magnitude.
16 . The computing system as recited in claim 15 , wherein the circuitry is further configured to replace the first set of weight values with a second set of weight values that have one or more magnitudes less than a corresponding one of the one or more magnitudes of the first set of weight values.
17 . The computing system as recited in claim 16 , wherein the second magnitude is less than the first magnitude.
18 . The computing system as recited in claim 15 , wherein the circuitry is further configured to:
receive a bit width P of an accumulator value used in the node; and
generate a weight magnitude limit as two to a power of P less N, wherein:
N is a bit width of input data values of the node; and
each of P and N is a positive, non-zero integer.
19 . The computing system as recited in claim 18 , wherein the circuitry is further configured to replace the first magnitude with the second magnitude, in response to determining a first sum of absolute values of the first magnitude and the one or more magnitudes of the first set of weight values is greater than the weight magnitude limit.
20 . The computing system as recited in claim 19 , wherein the circuitry is further configured to:
generate a difference between the first sum and the weight magnitude limit;
generate an average value by dividing the difference by a second sum of a number of weight values in the first set of weight values and one for the first weight value;
generate an adjustment value based at least in part on the average value; and
generate the second magnitude as an absolute value of the first magnitude less the adjustment value.