Neural network compression

1. A computer-implemented method for neural network compression, the method comprising:

receiving a neural network;

identifying a set of weights of the neural network;

determining initial values of a set of anchor points based on initial values of the set of weights of the neural network;

training the neural network by, at each of multiple training iterations, performing operations comprising:

adjusting current values of the set of weights of the neural network at the training iteration by backpropagating gradients of a loss function, wherein the loss function comprises:

a first loss function term based on a prediction accuracy of the neural network; and

a second loss function term based on a similarity of the current values of the set of weights of the neural network at the training iteration to current values of the set of anchor points at the training iteration; and

adjusting the current values of the set of anchor points at the training iteration based on the current values of the set of weights of the neural network at the training iteration, the adjusting comprising:

determining a set of quantiles of the current values of the set of weights of the neural network at the training iteration; and

setting, for each of one or more anchor points in the set of anchor points, a current value of the anchor point equal to a statistic of current values of a plurality of weights of the neural network that are included in an interval defined by a respective pair of quantiles from the set of quantiles;

quantizing the current values of the set of weights of the neural network, comprising, for each weight of the set of weights of the neural network:

determining an anchor point in the set of anchor points corresponding to the weight; and

setting a current value of the weight to a current value of the corresponding anchor point.

2. The method of claim 1 , wherein the second loss function term comprises a sum, over the set of weights, of a minimum distance between the current value of the weight and the current value of a corresponding anchor point.

3. The method of claim 1 , wherein determining the initial values of the set of anchor points based on the initial values of the set of weights comprises:

fitting a mixture model to a distribution of the initial values of the set of weights; and

determining the initial value of each anchor point in the set of anchor points based on parameters of components of the mixture model.

4. The method of claim 3 , wherein the mixture model is a Gaussian mixture model, and the initial value of each anchor point is determined based on mean parameters of one or more components of the Gaussian mixture model.

5. The method of claim 4 , wherein the components of the Gaussian mixture model are restricted to have a fixed standard deviation.

6. The method of claim 1 , wherein the neural network is pre-trained to perform a prediction task.

7. The method of claim 1 , wherein the neural network is a recurrent neural network.

8. The method of claim 1 , wherein the neural network is configured to process an image to generate a respective score for each category in a set of categories.

9. The method of claim 1 , wherein the neural network is configured to process text in one language to generate an output that characterizes a translation of the text into another language.

10. The method of claim 1 , wherein setting, for each of one or more anchor points in the set of anchor points, the current value of the anchor point equal to the statistic of the current values of the plurality of weights of the neural network that are included in the interval defined by the respective pair of quantiles from the set of quantiles comprises, for each of one or more anchor points:

setting the current value of the anchor point equal to a mean of the current values of the plurality of weights of the neural network that are included in the interval defined by the respective pair of quantiles from the set of quantiles.

11. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving a neural network;

identifying a set of weights of the neural network;

determining initial values of a set of anchor points based on initial values of the set of weights of the neural network;

training the neural network by, at each of multiple training iterations, performing operations comprising:

adjusting current values of the set of weights of the neural network at the training iteration by backpropagating gradients of a loss function, wherein the loss function comprises:

a first loss function term based on a prediction accuracy of the neural network; and

a second loss function term based on a similarity of the current values of the set of weights of the neural network at the training iteration to current values of the set of anchor points at the training iteration; and

adjusting the current values of the set of anchor points at the training iteration based on the current values of the set of weights of the neural network at the training iteration, the adjusting comprising:

determining a set of quantiles of the current values of the set of weights of the neural network at the training iteration; and

setting, for each of one or more anchor points in the set of anchor points, a current value of the anchor point equal to a statistic of current values of a plurality of weights of the neural network that are included in an interval defined by a respective pair of quantiles from the set of quantiles;

quantizing the current values of the set of weights of the neural network, comprising, for each weight of the set of weights of the neural network:

determining an anchor point in the set of anchor points corresponding to the weight; and

setting a current value of the weight to a current value of the corresponding anchor point.

12. The system of claim 11 , wherein the second loss function term comprises a sum, over the set of weights, of a minimum distance between the current value of the weight and the current value of a corresponding anchor point.

13. The system of claim 11 , wherein determining the initial values of the set of anchor points based on the initial values of the set of weights comprises:

fitting a mixture model to a distribution of the initial values of the set of weights; and

determining the initial value of each anchor point in the set of anchor points based on parameters of components of the mixture model.

14. The system of claim 13 , wherein the mixture model is a Gaussian mixture model, and the initial value of each anchor point is determined based on mean parameters of one or more components of the Gaussian mixture model.

15. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving a neural network;

identifying a set of weights of the neural network;

determining initial values of a set of anchor points based on initial values of the set of weights of the neural network;

training the neural network by, at each of multiple training iterations, performing operations comprising:

adjusting current values of the set of weights of the neural network at the training iteration by backpropagating gradients of a loss function, wherein the loss function comprises:

a first loss function term based on a prediction accuracy of the neural network; and

a second loss function term based on a similarity of the current values of the set of weights of the neural network at the training iteration to current values of the set of anchor points at the training iteration; and

adjusting the current values of the set of anchor points at the training iteration based on the current values of the set of weights of the neural network at the training iteration, the adjusting comprising:

determining a set of quantiles of the current values of the set of weights of the neural network at the training iteration; and

setting, for each of one or more anchor points in the set of anchor points, a current value of the anchor point equal to a statistic of current values of a plurality of weights of the neural network that are included in an interval defined by a respective pair of quantiles from the set of quantiles;

quantizing the current values of the set of weights of the neural network, comprising, for each weight of the set of weights of the neural network:

determining an anchor point in the set of anchor points corresponding to the weight; and

setting a current value of the weight to a current value of the corresponding anchor point.

16. The non-transitory computer storage media of claim 15 , wherein the second loss function term comprises a sum, over the set of weights, of a minimum distance between the current value of the weight and the current value of a corresponding anchor point.

17. The non-transitory computer storage media of claim 15 , wherein determining the initial values of the set of anchor points based on the initial values of the set of weights comprises:

fitting a mixture model to a distribution of the initial values of the set of weights; and

determining the initial value of each anchor point in the set of anchor points based on parameters of components of the mixture model.

18. The non-transitory computer storage media of claim 17 , wherein the mixture model is a Gaussian mixture model, and the initial value of each anchor point is determined based on mean parameters of one or more components of the Gaussian mixture model.

19. The non-transitory computer storage media of claim 18 , wherein the components of the Gaussian mixture model are restricted to have a fixed standard deviation.

20. The non-transitory computer storage media of claim 15 , wherein the neural network is pre-trained to perform a prediction task.