Training neural networks using synthetic gradients

1. A method performed by one or more computers for training a neural network to perform a task on training data comprising a plurality of training inputs by optimizing an objective function,

wherein the neural network is configured to receive a network input and to process the network input to generate a network output,

wherein the objective function is based on a measure of difference between (i) network outputs generated by the neural network by processing the training inputs and (ii) respective target outputs for each of the training inputs that are specified in the training data,

wherein the neural network comprises a plurality of subnetworks that include a first subnetwork followed by a second subnetwork and followed by a last subnetwork,

wherein the first subnetwork is configured to, during the processing of the network input by the neural network, receive a corresponding subnetwork input for the network input, process the corresponding subnetwork input for the network input to generate a corresponding subnetwork activation for the network input, and provide the corresponding subnetwork activation for the network input as input to the second subnetwork, and

wherein the method comprises, for each training input:

processing the training input using the neural network to generate a training model output for the training input, comprising:

inputting, into the first subnetwork, a corresponding subnetwork input for the training input;

processing the corresponding subnetwork input for the training input using the first subnetwork to generate a corresponding subnetwork activation for the training input in accordance with current values of parameters of the first subnetwork, and

providing the corresponding subnetwork activation for the training input as input to the second subnetwork;

determining a synthetic gradient for the first subnetwork that is an approximation of a gradient of the objective function with respect to parameters of the first subnetwork, wherein computing the gradient of the objective function with respect to parameters of the first subnetwork requires backward propagation of gradients from the last subnetwork to the second subnetwork and from the second subnetwork into the first subnetwork, and wherein determining the synthetic gradient that is an approximation of the gradient comprises processing the corresponding subnetwork activation using a synthetic gradient model for the first subnetwork in accordance with current values of parameters of the synthetic gradient model, wherein the synthetic gradient model for the first subnetwork is configured to process the corresponding subnetwork activation in accordance with the current values of the parameters of the synthetic gradient model to generate the synthetic gradient for the first subnetwork;

updating the current values of the parameters of the first subnetwork using the synthetic gradient;

determining a target gradient of the objective function with respect to the parameters of the first subnetwork by backpropagating a gradient of the objective function with respect to parameters of the second subnetwork from the second subnetwork into the first subnetwork; and

training the synthetic gradient model to update current values of the parameters of the synthetic gradient model based on an error between (i) the target gradient of the objective function with respect to the parameters of the first subnetwork determined by backpropagating a gradient of the objective function with respect to parameters of the second subnetwork from the second subnetwork into the first subnetwork and (ii) the synthetic gradient for the first subnetwork that is an approximation of the gradient of the objective function with respect to the parameters of the first subnetwork.

2. The method of claim 1 , wherein the synthetic gradient model is a different neural network from each of the neural network, the first subnetwork, and the second subnetwork.

3. The method of claim 1 , wherein the gradient of the objective function with respect to parameters of the second subnetwork from the second subnetwork into the first subnetwork is:

an actual gradient of the objective function that has been backpropagated through the neural network; or

a synthetic gradient for the second subnetwork.

4. The method of claim 1 , wherein the first subnetwork comprises multiple neural network layers, and wherein updating the current values of the parameters of the first subnetwork using the synthetic gradient comprises:

backpropagating the synthetic gradient through the first subnetwork to update the current values of the parameters of the first subnetwork.

5. The method of claim 1 , wherein the neural network is a feedforward neural network, the first subnetwork is a first neural network layer, and the second subnetwork is a second neural network layer.

6. The method of claim 1 , wherein the neural network is an unrolled recurrent neural network, the first subnetwork is the recurrent neural network at a first time step, and the second subnetwork is the recurrent neural network at a second time step.

7. The method of claim 6 , further comprising:

determining at least one future synthetic gradient of the objective function for the first subnetwork by processing the corresponding subnetwork activation for the training input using the synthetic gradient model for the first subnetwork in accordance with current values of parameters of the synthetic gradient model; and

updating the current values of the parameters of the synthetic gradient model based on an error between each future synthetic gradient and a corresponding target future gradient.

8. The method of claim 1 , wherein updating the current values of the parameters of the first subnetwork using the synthetic gradient comprises updating the current values of the parameters using the synthetic gradient in place of an actual backpropagated gradient.

9. The method of claim 1 , wherein updating the current values of the parameters of the first subnetwork using the synthetic gradient comprises updating the current values of the parameters using the synthetic gradient asynchronously from updating current values of the parameters of the second subnetwork.

10. The method of claim 1 , wherein the first subnetwork is implemented on one computing device and the second subnetwork is implemented on a different computing device; and wherein:

the training is part of a distributed machine learning training process that distributes the training across multiple computing devices.

11. The method of claim 1 , wherein the corresponding subnetwork input for the training input is a synthetic subnetwork input, and wherein the method further comprises:

processing the training input using a synthetic input model that is configured to process the training input to generate the synthetic subnetwork input.

12. The method of claim 1 , wherein the subnetwork input for the training input is an actual subnetwork input.

13. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network to perform a task on training data comprising a plurality of training inputs by optimizing an objective function,

wherein the neural network is configured to receive a network input and to process the network input to generate a network output,

wherein the objective function is based on a measure of difference between (i) network outputs generated by the neural network by processing the training inputs and (ii) respective target outputs for each of the training inputs that are specified in the training data,

wherein the neural network comprises a plurality of subnetworks that include a first subnetwork followed by a second subnetwork and followed by a last subnetwork,

wherein the first subnetwork is configured to, during the processing of the network input by the neural network, receive a corresponding subnetwork input for the network input, process the corresponding subnetwork input for the network input to generate a corresponding subnetwork activation for the network input, and provide the corresponding subnetwork activation for the network input as input to the second subnetwork, and

wherein the operations comprise, for each training input:

processing the training input using the neural network to generate a training model output for the training input, comprising:

inputting, into the first subnetwork, a corresponding subnetwork input for the training input;

processing the corresponding subnetwork input for the training input using the first subnetwork to generate a corresponding subnetwork activation for the training input in accordance with current values of parameters of the first subnetwork, and

providing the corresponding subnetwork activation for the training input as input to the second subnetwork;

determining a synthetic gradient for the first subnetwork that is an approximation of a gradient of the objective function with respect to parameters of the first subnetwork, wherein computing the gradient of the objective function with respect to parameters of the first subnetwork requires backward propagation of gradients from the last subnetwork to the second subnetwork and from the second subnetwork into the first subnetwork, and wherein determining the synthetic gradient that is an approximation of the gradient comprises processing the corresponding subnetwork activation using a synthetic gradient model for the first subnetwork in accordance with current values of parameters of the synthetic gradient model, wherein the synthetic gradient model for the first subnetwork is configured to process the corresponding subnetwork activation in accordance with the current values of the parameters of the synthetic gradient model to generate the synthetic gradient for the first subnetwork;

updating the current values of the parameters of the first subnetwork using the synthetic gradient;

determining a target gradient of the objective function with respect to the parameters of the first subnetwork by backpropagating a gradient of the objective function with respect to parameters of the second subnetwork from the second subnetwork into the first subnetwork; and

training the synthetic gradient model to update current values of the parameters of the synthetic gradient model based on an error between (i) the target gradient of the objective function with respect to the parameters of the first subnetwork determined by backpropagating a gradient of the objective function with respect to parameters of the second subnetwork from the second subnetwork into the first subnetwork and (ii) the synthetic gradient for the first subnetwork that is an approximation of the gradient of the objective function with respect to the parameters of the first subnetwork.

14. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a neural network to perform a task on training data comprising a plurality of training inputs by optimizing an objective function,

wherein the neural network is configured to receive a network input and to process the network input to generate a network output,

wherein the objective function is based on a measure of difference between (i) network outputs generated by the neural network by processing the training inputs and (ii) respective target outputs for each of the training inputs that are specified in the training data,

wherein the neural network comprises a plurality of subnetworks that include a first subnetwork followed by a second subnetwork and followed by a last subnetwork,

wherein the first subnetwork is configured to, during the processing of the network input by the neural network, receive a corresponding subnetwork input for the network input, process the corresponding subnetwork input for the network input to generate a corresponding subnetwork activation for the network input, and provide the corresponding subnetwork activation for the network input as input to the second subnetwork, and

wherein the operations comprise, for each training input:

processing the training input using the neural network to generate a training model output for the training input, comprising:

inputting, into the first subnetwork, a corresponding subnetwork input for the training input;

processing the corresponding subnetwork input for the training input using the first subnetwork to generate a corresponding subnetwork activation for the training input in accordance with current values of parameters of the first subnetwork, and

providing the corresponding subnetwork activation for the training input as input to the second subnetwork;

determining a synthetic gradient for the first subnetwork that is an approximation of a gradient of the objective function with respect to parameters of the first subnetwork, wherein computing the gradient of the objective function with respect to parameters of the first subnetwork requires backward propagation of gradients from the last subnetwork to the second subnetwork and from the second subnetwork into the first subnetwork, and wherein determining the synthetic gradient that is an approximation of the gradient comprises processing the corresponding subnetwork activation using a synthetic gradient model for the first subnetwork in accordance with current values of parameters of the synthetic gradient model, wherein the synthetic gradient model for the first subnetwork is configured to process the corresponding subnetwork activation in accordance with the current values of the parameters of the synthetic gradient model to generate the synthetic gradient for the first subnetwork;

updating the current values of the parameters of the first subnetwork using the synthetic gradient;

determining a target gradient of the objective function with respect to the parameters of the first subnetwork by backpropagating a gradient of the objective function with respect to parameters of the second subnetwork from the second subnetwork into the first subnetwork; and

training the synthetic gradient model to update current values of the parameters of the synthetic gradient model based on an error between (i) the target gradient of the objective function with respect to the parameters of the first subnetwork determined by backpropagating a gradient of the objective function with respect to parameters of the second subnetwork from the second subnetwork into the first subnetwork and (ii) the synthetic gradient for the first subnetwork that is an approximation of the gradient of the objective function with respect to the parameters of the first subnetwork.

15. The system of claim 14 , wherein the synthetic gradient model is a different neural network from each of the neural network, the first subnetwork, and the second subnetwork.

16. The system of claim 14 , wherein the gradient of the objective function with respect to parameters of the second subnetwork from the second subnetwork into the first subnetwork is:

an actual gradient of the objective function that has been backpropagated through the neural network; or

a synthetic gradient for the second subnetwork.

17. The system of claim 14 , wherein the first subnetwork comprises multiple neural network layers, and wherein updating the current values of the parameters of the first subnetwork using the synthetic gradient comprises:

backpropagating the synthetic gradient through the first subnetwork to update the current values of the parameters of the first subnetwork.

18. The system of claim 14 , wherein the neural network is a feedforward neural network, the first subnetwork is a first neural network layer, and the second subnetwork is a second neural network layer.