Training neural networks on partitioned training data

1. A method for training a neural network, the method comprising:

obtaining, by one or more computers, partitioned training data for use in training the neural network, wherein the partitioned training data comprises a plurality of training items each of which is assigned to a respective one of a plurality of partitions;

obtaining, by the one or more computers, an input specifying a difficulty order for the partitions; and

training, by the one or more computers, the neural network on each of the partitions according to the difficulty order and starting from a first partition in the difficulty order and ending with a last partition in the difficulty order, wherein, for each particular partition of the plurality of partitions, training the neural network comprises:

generating a sequence of training items for the particular partition, comprising:

selecting a plurality of training items from among all of the training items in all of the plurality of partitions, and

generating a sequence of training items that includes the training items selected from among all of the training items in all of the plurality of positions inserted between training items selected only from the plurality of training items in the particular partition; and

training the neural network on the sequence of training items for the partition.

2. The method of claim 1 , wherein the training items selected from the training items in all of the partitions are interspersed at predetermined regular intervals in the sequence for the particular partition.

3. The method of claim 1 , wherein the training items selected from the training items in the particular partition are a majority of the training items in the sequence for the particular partition.

4. The method of claim 1 , wherein selecting the plurality of training items from among all of the training items in all of the plurality of partitions comprises, for each selected training item:

randomly selecting a partition from the plurality of partitions; and

randomly selecting the training item from among the training items in the randomly selected partition.

5. The method of claim 1 , wherein selecting the plurality of training items from among all of the training items in all of the plurality of partitions comprises, for each selected training item selecting the training item randomly from among the plurality of training items.

6. The method of claim 1 , wherein the training items selected only from the plurality of training items in the particular partition comprises are selected randomly from the training items in the particular partition.

7. The method of claim 1 , wherein, for each particular partition of the plurality of partitions, training the neural network further comprises:

while training the neural network on the particular partition, determining whether a performance of the neural network has stopped improving; and

refraining from training the neural network further on the particular partition in response to determining that the performance of the neural network has stopped improving.

8. The method of claim 7 , wherein determining that the performance of the neural network has stopped improving comprises determining that a reduction in an error measure for the training has become lower than a threshold.

9. The method of claim 1 , wherein the neural network is a recurrent neural network.

10. The method of claim 9 , wherein the recurrent neural network is a long short term (LSTM) neural network.

11. The method of claim 9 , wherein each of the training items is a sequence of code from a respective computer program and wherein the recurrent neural network is configured to process the sequence of code and output a predicted output of the computer program.

12. The method of claim 11 , wherein the difficulty order is based at least in part on lengths of values that appear in the code.

13. The method of claim 11 , wherein difficulty order is based at least in part on levels of nesting that appear in the code.

14. A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining partitioned training data for use in training the neural network, wherein the partitioned training data comprises a plurality of training items each of which is assigned to a respective one of a plurality of partitions;

obtaining an input specifying a difficulty order for the partitions; and

training the neural network on each of the partitions according to the difficulty order and starting from a first partition in the difficulty order and ending with a last partition in the difficulty order, wherein, for each particular partition of the plurality of partitions, training the neural network comprises:

generating a sequence of training items for the particular partition, comprising:

selecting a plurality of training items from among all of the training items in all of the plurality of partitions, and

generating a sequence of training items that includes the training items selected from among all of the training items in all of the plurality of positions inserted between training items selected only from the plurality of training items in the particular partition; and

training the neural network on the sequence of training items for the partition.

15. The system of claim 14 , wherein the training items selected from the training items in all of the partitions are interspersed at predetermined regular intervals in the sequence for the particular partition.

16. The system of claim 14 , wherein the training items selected from the training items in the particular partition are a majority of the training items in the sequence for the particular partition.

17. The system of claim 14 , wherein selecting the plurality of training items from among all of the training items in all of the plurality of partitions comprises, for each selected training item selecting the training item randomly from among the plurality of training items.

18. The system of claim 14 , wherein, for each particular partition of the plurality of partitions, training the neural network further comprises:

while training the neural network on the particular partition, determining whether a performance of the neural network has stopped improving; and

refraining from training the neural network further on the particular partition in response to determining that the performance of the neural network has stopped improving.

19. A computer program product encoded on one or more non-transitory computer readable storage media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining partitioned training data for use in training the neural network, wherein the partitioned training data comprises a plurality of training items each of which is assigned to a respective one of a plurality of partitions;

obtaining an input specifying a difficulty order for the partitions; and

training the neural network on each of the partitions according to the difficulty order and starting from a first partition in the difficulty order and ending with a last partition in the difficulty order, wherein, for each particular partition of the plurality of partitions, training the neural network comprises:

generating a sequence of training items for the particular partition, comprising:

selecting a plurality of training items from among all of the training items in all of the plurality of partitions, and

generating a sequence of training items that includes the training items selected from among all of the training items in all of the plurality of positions inserted between training items selected only from the plurality of training items in the particular partition; and

training the neural network on the sequence of training items for the partition.