Dithered quantization of parameters during training with a machine learning tool
A machine learning tool uses dithered quantization of parameters during training of a machine learning model such as a neural network. The machine learning tool receives training data and initializes certain parameters of the machine learning model (e.g., weights for connections between nodes of a neural network, biases for nodes). The machine learning tool trains the parameters in one or more iterations based on the training data. In particular, in a given iteration, the machine learning tool applies the machine learning model to at least some of the training data and, based at least in part on the results, determines parameter updates to the parameters. The machine learning tool updates the parameters using the parameter updates and a dithered quantizer function, which can add random values before a rounding or truncation operation.
1 . A method, implemented in a computer system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, the method comprising:
receiving training data;
initializing parameters of a machine learning model;
training at least some of the parameters in multiple iterations based on the training data, including, in each of the multiple iterations during the training:
applying the machine learning model to at least some of the training data;
based at least in part on results of the applying the machine learning model, determining a measure of loss as a difference between an output of the machine learning model for the at least some of the training data and an expected output of the machine learning model for the at least some of the training data;
based at least in part on the measure of loss for the at least some of the training data, determining a set of parameter updates to the at least some of the parameters;
combining predetermined respective dither values from a random value generator function with respective combinations of respective parameter updates of the set of parameter updates and respective starting parameter values of respective parameters of the at least some of the parameters used during the applying the machine learning model to at least some of the training data, wherein at least a portion of the respective predetermined dither values differ from one another, to provide respective quantizer function operands; and
after the combining the predetermined respective dither values, quantizing the respective quantizer function operands using a quantizer function to provide one or more updated parameter values for at least a portion of the at least some of the parameters, wherein the one or more updated parameter values are used in at least one subsequent iteration of the multiple iterations.
2 . The method of claim 1 , wherein the machine learning model is a neural network having multiple layers, each of the multiple layers including one or more nodes, and wherein the parameters include one or more of:
weights for connections between the one or more nodes;
biases for at least some of the one or more nodes;
a count of the multiple layers;
for each of the multiple layers, a count of nodes in the layer; and
a control parameter for batch normalization.
3 . The method of claim 1 , wherein the combining and the quantizing are implemented as, where w t is a starting parameter value, Δw t is a parameter update, and d is a dither value:
round(w t +Δw t +d), wherein round( ) is a rounding function;
round((w t +Δw t +d)/p)×p, wherein p is a level of precision;
floor(w t +Δw t +d), wherein floor( ) is a floor function;
floor((w t +Δw t +d)/p)×p, wherein p is a level of precision; or
quant(w t +Δw t +d), wherein quant( ) is a quantizer function.
4 . The method of claim 1 , wherein the at least some of the parameters are in a floating-point format having a given level of precision for mantissa values, and wherein the combining applies dithering values d associated with the at least some of the parameters, respectively, by adding mantissa values at a higher level of precision before rounding or truncating to a nearest mantissa value for the given level of precision.
5 . The method of claim 1 , wherein the at least some of the parameters are in an integer format or fixed-point format, and wherein the combining applies dithering values d associated with the at least some of the parameters, respectively, by adding values at a higher level of precision before rounding or truncating to a nearest integer value.
6 . The method of claim 1 , wherein the quantizing is implemented using a rounding function, wherein the at least some of the parameters are in a format having a given level of precision after quantization is applied, and wherein dithering values d for the combining are selected in a range of −0.5 to 0.5 of an increment of a value at the given level of precision after quantization.
7 . The method of claim 1 , wherein the quantizing is implemented using a floor function, wherein the at least some of the parameters are in a format having a given level of precision after quantization is applied, and wherein dithering values d for the combining are selected in a range of 0.0 to 1.0 of an increment of a value at the given level of precision after quantization.
8 . The method of claim 1 , wherein the determining the set of parameter updates, Δw t , includes multiplying unscaled parameter updates Δw′ t by a learning rate η, as Δw t =η×Δw′ t .
9 . The method of claim 1 , wherein the training the at least some of the parameters further includes:
converting values, including the at least some of the parameters, from a first format to a second format, the second format having a lower precision than the first format, wherein the first format is a first floating-point format having m 1 bits of precision for mantissa values and e1 bits of precision for exponent values, wherein the second format is a second floating-point format:
having m 2 bits of precision for mantissa values and e 2 bits of precision for exponent values, wherein m 1 >m 2 , and wherein e 1 >e 2 ; or
having a shared exponent value.
10 . The method of claim 1 , wherein the machine learning model is a deep neural network, a support vector machine, a Bayesian network, a decision tree, or a linear classifier.
11 . The method of claim 1 , wherein at least a portion of the respective dither values are based at least in part on output of a random number generator, and wherein the respective dither values are random values having a power spectrum of white noise or blue noise.
12 . The method of claim 1 , wherein the quantizer function is the same in each of the multiple iterations.
13 . The method of claim 1 , wherein the training data includes multiple examples, each of the multiple examples having one or more attributes and a label, and wherein each of the multiple iterations uses a single example or mini-batch of examples randomly selected from among any remaining examples, for an epoch, of the multiple examples.
14 . The method of claim 1 , wherein the initializing the parameters includes:
for an initial iteration of an initial epoch, setting the parameters to random values.
15 . A computer system comprising:
a buffer, in memory of the computer system, configured to receive training data; and
a machine learning tool, implemented with one or more processors of the computer system, configured to perform operations comprising:
initializing parameters of a machine learning model;
training at least some of the parameters in multiple iterations based on the training data, including, in each of the multiple iterations during the training:
applying the machine learning model to at least some of the training data;
based at least in part on results of the applying the machine learning model, determining a measure of loss as a difference between an output of the machine learning model for the at least some of the training data and an expected output of the machine learning model for the at least some of the training data;
based at least in part on the measure of loss for the at least some of the training data, determining a set of parameter updates to the at least some of the parameters;
combining predetermined respective dither values from a random value generator function with respective combinations of respective parameter updates of the set of parameter updates and respective starting parameter values of respective parameters of the at least some of the parameters used during the applying the machine learning model to at least some of the training data, wherein at least a portion of the respective predetermined dither values differ from one another, to provide respective quantizer function operands;
after the combining the predetermined respective dither values, quantizing the respective quantizer function operands using a quantizer function to provide one or more updated parameter values for at least a portion of the at least some of the parameters, wherein the one or more updated parameter values are used in at least one subsequent iteration of training of the machine learning model; and
a buffer, in memory of the computer system, configured to store the parameters.
16 . One or more computer-readable media having stored thereon computer-executable instructions for causing one or more processing units, when programmed thereby, to perform operations comprising:
receiving training data;
initializing parameters of a machine learning model;
training at least some of the parameters in multiple iterations based on the training data, including, in each of the multiple iterations during the training:
applying the machine learning model to at least some of the training data;
based at least in part on results of the applying the machine learning model, determining a measure of loss as a difference between an output of the machine learning model for the at least some of the training data and an expected output of the machine learning model for the at least some of the training data;
based at least in part on the measure of loss for the at least some of the training data, determining a set of parameter updates to the at least some of the parameters;
combining predetermined respective dither values from a random value generator function with respective combinations of respective parameter updates of the set of parameter updates and respective starting parameter values of respective parameters of the at least some of the parameters used during the applying the machine learning model to at least some of the training data, wherein at least a portion of the respective predetermined dither values differ from one another, to provide respective quantizer function operands; and
after the combining the predetermined respective dither values, quantizing the respective quantizer function operands using a quantizer function to provide one or more updated parameter values for at least a portion of the at least some of the parameters, wherein the one or more updated parameter values are used in at least one subsequent iteration of training of the machine learning model.
17 . The method of claim 1 , wherein (1) the quantizer function is different between at least a portion of the multiple iterations, or (2) the quantizer function is not applied during one or more iterations of the multiple iterations.
18 . The computer system of claim 15 , wherein (1) the dithered quantizer function is different between at least a portion of the multiple iterations, or (2) the dithered quantization function is not applied during one or more iterations of the multiple iterations.
19 . The one or more computer-readable storage media of claim 16 , wherein (1) the dithered quantizer function is different between at least a portion of the multiple iterations, or (2) the dithered quantization function is not applied during one or more iterations of the multiple iterations.
20 . The method of claim 1 , wherein the at least a portion of the respective dither values are independently selected.