Training neural networks with reinitialization

1 . A method performed by one or more computers, the method comprising:

training a neural network comprising a plurality of neural network layers each having a respective plurality of weights, wherein at least a subset of the neural network layers are arranged into a sequence of layer blocks, and wherein the training comprises re-initializing weights of one or more of the layer blocks at each of a sequence of training time steps to improve generalization of the neural network to new inputs after training, comprising:

at each training time step in the sequence of a plurality of training time steps:

identifying current values of the weights of the plurality of neural network layers as of the training time step;

selecting one of the layer blocks;

determining whether the selected layer block has previously been selected at any earlier training time step in the sequence;

in response to determining that the selected layer block has not previously been selected, inserting a normalization layer into the neural network immediately after the selected layer block;

generating new values for the weights of the plurality of neural network layers, comprising:

re-initializing the values of the weights of at least the neural network layers in the layer blocks that are after the selected layer block in the sequence without re-initializing the current values of the weights of the neural network layers in the selected layer block and the neural network layers in any layer block that is before the selected layer block in the sequence; and

training the neural network starting from the new values for the weights of the plurality of neural network layers to determine updated values for the weights of the plurality of neural network layers.

2 . The method of claim 1 , wherein the normalization layer normalizes inputs to the normalization layer without applying any learned parameters that are adjusted during the training of the neural network.

3 . The method of claim 1 , further comprising, in response to determining that the selected layer block has previously been selected, updating the normalization statistics used by the normalization layer to normalize the outputs of the selected block.

4 . The method of claim 1 , wherein selecting one of the layer blocks comprises:

determining whether criteria for selecting a new layer block are satisfied; and

when the criteria are not satisfied, selecting the same layer block as was selected at the preceding training time step.

5 . The method of claim 4 , wherein the criteria are satisfied only when the layer block that was selected at the preceding training time step has already been selected at a threshold number of training time steps.

6 . The method of claim 1 , wherein selecting one of the layer blocks comprises:

determining whether criteria for selecting a new layer block are satisfied; and

when the criteria are satisfied, selecting the layer block that is immediately after the layer block that was selected at the preceding training time step in the sequence in the neural network.

7 . The method of claim 1 , wherein the neural network comprises one or more neural network layers that are after the last layer block in the sequence and wherein generating new values for the weights of the plurality of neural network layers comprises:

re-initializing the values of the weights of at least the neural network layers that are after the last layer block in the sequence.

8 . The method of claim 1 , further comprising, prior to the first training time step:

initializing the values of the weights of the neural networks layers in the neural network; and

training the neural network to generate the current values of the weights of the neural network for the first training time step.

9 . The method of claim 1 , wherein the neural network is trained on the same training data at all of the training time steps.

10 . The method of claim 1 , wherein training the neural network comprises training the neural network to convergence.

11 . The method of claim 1 , wherein generating new values comprises:

rescaling the current values of the weights of the neural network layers in the layer block and the neural network layers in any layer block that is before the selected layer block in the sequence.

12 . One or more non-transitory computer-readable media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

training a neural network comprising a plurality of neural network layers each having a respective plurality of weights, wherein at least a subset of the neural network layers are arranged into a sequence of layer blocks, and wherein the training comprises re-initializing weights of one or more of the layer blocks at each of a sequence of training time steps to improve generalization of the neural network to new inputs after training, comprising:

at each training time step in the sequence of a plurality of training time steps:

identifying current values of the weights of the plurality of neural network layers as of the training time step;

selecting one of the layer blocks;

determining whether the selected layer block has previously been selected at any earlier training time step in the sequence;

in response to determining that the selected layer block has not previously been selected, inserting a normalization layer into the neural network immediately after the selected layer block;

generating new values for the weights of the plurality of neural network layers, comprising:

re-initializing the values of the weights of at least the neural network layers in the layer blocks that are after the selected layer block in the sequence without re-initializing the current values of the weights of the neural network layers in the selected layer block and the neural network layers in any layer block that is before the selected layer block in the sequence; and

training the neural network starting from the new values for the weights of the plurality of neural network layers to determine updated values for the weights of the plurality of neural network layers.

13 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

training a neural network comprising a plurality of neural network layers each having a respective plurality of weights, wherein at least a subset of the neural network layers are arranged into a sequence of layer blocks, and wherein the training comprises re-initializing weights of one or more of the layer blocks at each of a sequence of training time steps to improve generalization of the neural network to new inputs after training, comprising:

at each training time step in the sequence of a plurality of training time steps:

identifying current values of the weights of the plurality of neural network layers as of the training time step;

selecting one of the layer blocks;

determining whether the selected layer block has previously been selected at any earlier training time step in the sequence;

in response to determining that the selected layer block has not previously been selected, inserting a normalization layer into the neural network immediately after the selected layer block;

generating new values for the weights of the plurality of neural network layers, comprising:

re-initializing the values of the weights of at least the neural network layers in the layer blocks that are after the selected layer block in the sequence without re-initializing the current values of the weights of the neural network layers in the selected layer block and the neural network layers in any layer block that is before the selected layer block in the sequence; and

training the neural network starting from the new values for the weights of the plurality of neural network layers to determine updated values for the weights of the plurality of neural network layers.

14 . The system of claim 13 , wherein the normalization layer normalizes inputs to the normalization layer without applying any learned parameters that are adjusted during the training of the neural network.

15 . The system of claim 13 , the operations further comprising, in response to determining that the selected layer block has previously been selected, updating the normalization statistics used by the normalization layer to normalize the outputs of the selected block.

16 . The system of claim 13 , wherein selecting one of the layer blocks comprises:

determining whether criteria for selecting a new layer block are satisfied; and

when the criteria are not satisfied, selecting the same layer block as was selected at the preceding training time step.

17 . The system of claim 13 , wherein selecting one of the layer blocks comprises:

determining whether criteria for selecting a new layer block are satisfied; and

when the criteria are satisfied, selecting the layer block that is immediately after the layer block that was selected at the preceding training time step in the sequence in the neural network.

18 . The system of claim 17 , wherein the criteria are satisfied only when the layer block that was selected at the preceding training time step has already been selected at a threshold number of training time steps.