Training network with discrete weight values
Some embodiments provide an electronic device that includes a set of processing units and a set of machine-readable media. The set of machine-readable media stores sets of instructions for applying a network of computation nodes to an input received by the device. The set of machine-readable media stores at least two sets of machine-trained parameters for configuring the network for different types of inputs. A first of the sets of parameters is used for applying the network to a first type of input and a second of the sets of parameters is used for applying the network to a second type of input.
1. A method for training a plurality of weights of a neural network comprising a plurality of layers, the method comprising:
propagating a plurality of inputs through the plurality of layers of the neural network to generate a plurality of outputs, each input having a corresponding expected output;
calculating a value for a loss function that comprises (i) a loss term that measures, for each input, a difference between the generated output and the expected output for the input and (ii) a constraint term that penalizes weights for differentiating from a ternary set of allowed values for the weights, wherein the ternary set of allowed values for each weight comprises zero, a positive value for a layer to which the weight belongs, and a negation of the positive value for the layer to which the weight belongs; and
using the calculated value for the loss function comprising the loss term and the constraint term to train the weights of the neural network, wherein constraining the weights to the ternary set causes weight data for the network to be represented using two or fewer bits for each weight and stored on a processor without the use of a separate memory, for a device in which the neural network is embedded, wherein the processor for the device (i) stores each weight as a ternary value of the set {0, 1, −1} and (ii) computes dot products involving the weights using the ternary value and multiplies the dot products by the positive value for the layer to which the weight belongs when executing the neural network.
2. The method of claim 1 , wherein the constraint term is a continuously-differentiable function of each of the weights that is equal to zero for a particular weight when the particular weight is equal to any one of the ternary set of allowed values for the particular weight.
3. The method of claim 1 , wherein the ternary set of allowed values for a first layer is different than the ternary set of allowed values for a second layer.
4. The method of claim 1 , wherein propagating the plurality of inputs through the plurality of layers of the neural network comprises rounding each respective weight to one of the respective allowed weight values for the respective weight value in order to propagate the plurality of inputs.
5. The method of claim 1 further comprising iteratively propagating inputs through the neural network, calculating a new value for the loss function, and using the new calculated value to train the weights of the neural network.
6. The method of claim 5 , wherein a size of the constraint term is increased relative to the loss term during later iterations such that the constraint term has a larger effect on the training during the later iterations.
7. The method of claim 1 , wherein the neural network is trained to perform a specific function when embedded in the device.
8. The method of claim 1 further comprising imposing an additional constraint on the weights of the network that at least a particular percentage of the weights with ternary values be equal to zero.
9. The method of claim 1 , wherein using the calculated value for the loss function to train the weights of the neural network comprises:
backpropagating the calculated value through the neural network to determine, for each weight, a rate of change in the calculated value relative to a rate of change in the weight; and
modifying each particular weight according to the determined rate of change for the particular weight.
10. A non-transitory machine-readable medium storing a program which when executed by at least one processing unit trains a plurality of weights of a neural network comprising a plurality of layers, the program comprising sets of instructions for:
propagating a plurality of inputs through the plurality of layers of the neural network to generate a plurality of outputs, each input having a corresponding expected output;
calculating a value for a loss function that comprises (i) a loss term that measures, for each input, a difference between the generated output and the expected output for the input and (ii) a constraint term that penalizes weights for differentiating from a ternary set of allowed values for the weights, wherein the ternary set of allowed values for each weight comprises zero, a positive value for a layer to which the weight belongs, and a negation of the positive value for the layer to which the weight belongs; and
using the calculated value for the loss function comprising the loss term and the constraint term to train the weights of the neural network, wherein constraining the weights to the ternary set causes weight data for the network to be represented using two or fewer bits for each weight and stored on a processor without the use of a separate memory, for a device in which the neural network is embedded, wherein the processor for the device (i) stores each weight as a ternary value of the set {0, 1, −1} and (ii) computes dot products involving the weights using the ternary value and multiplies the dot products by the positive value for the layer to which the weight belongs when executing the neural network.
11. The non-transitory machine-readable medium of claim 10 , wherein the constraint term is a continuously-differentiable function of each of the weights that is equal to zero for a particular weight when the particular weight is equal to any one of the ternary set of allowed values for the particular weight.
12. The non-transitory machine-readable medium of claim 10 , wherein the ternary set of allowed values for a first layer is different than the ternary set of allowed values for a second layer.
13. The non-transitory machine-readable medium of claim 10 , wherein the set of instructions for propagating the plurality of inputs through the plurality of layers of the neural network comprises a set of instructions for rounding each respective weight to one of the respective allowed weight values for the respective weight value in order to propagate the plurality of inputs.
14. The non-transitory machine-readable medium of claim 10 , wherein the program further comprises sets of instructions for iteratively propagating inputs through the neural network, calculating a new value for the loss function, and using the new calculated value to train the weights of the neural network.
15. The non-transitory machine-readable medium of claim 14 , wherein a size of the constraint term is increased relative to the loss term during later iterations such that the constraint term has a larger effect on the training during the later iterations.
16. The non-transitory machine-readable medium of claim 10 , wherein the neural network is trained to perform a specific function when embedded in the device.
17. The non-transitory machine-readable medium of claim 10 , wherein the program further comprises a set of instructions for imposing an additional constraint on the weights of the network that at least a particular percentage of the weights with ternary values be equal to zero.
18. The non-transitory machine-readable medium of claim 10 , wherein the set of instructions for using the calculated value for the loss function to train the weights of the neural network comprises sets of instructions for:
backpropagating the calculated value through the neural network to determine, for each weight, a rate of change in the calculated value relative to a rate of change in the weight; and
modifying each particular weight according to the determined rate of change for the particular weight.