Training neural networks using data augmentation policies

1. A method comprising:

obtaining a training data set for training a machine learning model having a plurality of network parameters to perform a particular machine learning task, the training data set comprising a plurality of training inputs;

determining a plurality of data augmentation policies, wherein each data augmentation policy defines a procedure for processing a training input to generate a transformed training input, wherein the procedure comprises selecting a sequence of one or more transformation operations to be applied to the training input, and wherein the determining comprises, for each data augmentation policy:

determining a value for a first hyperparameter that specifies a same number of transformation operations in each sequence of transformation operations selected for transforming a respective training input, and

determining a respective value for each of one or more second hyperparameters, wherein the second hyperparameters specify a same magnitude schedule for determining a magnitude for each transformation operation in each sequence of transformation operations selected for transforming a respective training input;

for each data augmentation policy, training the machine learning model using the data augmentation policy, comprising:

determining an augmented training data set by processing the training inputs in the training data set to generate transformed training inputs in accordance with the data augmentation policy,

processing a particular transformed training input in the augmented training data set using current values for the plurality of network parameters of the machine learning model to generate a training output, and

generating a parameter update for the plurality of parameters of the machine learning model based on an error in the training output;

determining, for each data augmentation policy, a quality measure of the machine learning model that has been trained using the data augmentation policy, wherein the quality measure of the machine learning model represents a performance of the machine learning model on the particular machine learning task as a result of training the machine learning model using the data augmentation policy; and

selecting a final data augmentation policy based using the quality measures of the machine learning models.

2. The method of claim 1 , further comprising:

providing the machine learning model that was trained using the selected final data augmentation policy as a final trained machine learning model for the particular machine learning task.

3. The method of claim 1 , further comprising:

augmenting the training data set according to the final data augmentation policy to generate a final augmented training data set; and

training the machine learning model on the final augmented training data set.

4. The method of claim 1 , wherein the magnitude schedule is defined by a single second hyperparameter that specifies a fixed magnitude throughout training.

5. The method of claim 1 , wherein the second hyperparameters comprise:

a first value that specifies the magnitude of transformation operations in a first training time period, and

a second value that specifies an amount by which the magnitude of transformation operations will change in one or more subsequent training time periods.

6. The method of claim 1 , wherein:

the second hyperparameters comprise i) a first value that specifies a minimum magnitude of transformation operations, and ii) a second value that specifies a maximum magnitude of transformation operations; and

processing a training input to generate a transformed training input comprises, for each transformation operation in the sequence of transformation operations selected for transforming the training input, randomly sampling a magnitude for the transformation operation from a range of magnitudes defined by the minimum magnitude and the maximum magnitude.

7. The method of claim 1 , wherein:

the second hyperparameters comprise i) a first value that specifies a minimum magnitude of transformation operations, ii) a second value that specifies a maximum magnitude of transformation operations in a first training time period, and iii) a third value that specifies an amount by which the maximum magnitude of transformation operations will change in one or more subsequent training time periods; and

processing a training input to generate a transformed training input in a given training time period comprises, for each transformation operation in the sequence of transformation operations selected for transforming the training input, randomly sampling a magnitude for the transformation operation from a range of magnitudes defined by the minimum magnitude and the maximum magnitude corresponding to the given training time period.

8. The method of claim 1 , wherein each transformation operation in each sequence of transformation operations is selected from a plurality of candidate transformation operations.

9. The method of claim 8 , wherein the training inputs are images and wherein the plurality of candidate transformation operations comprise one or more of:

a rotation operation;

a posterizing operation;

a sharpness operation;

a translation operation;

an auto-contrast operation;

a contrast operation;

a solarization operation;

a shearing operation;

a color operation;

a brightness operation;

a flipping operation;

a scale jittering operation;

an equalization operation;

an identity operation; or

a random cropping operation.

10. The method of claim 8 , wherein determining a plurality of data augmentation policies further comprises, for each data augmentation policy:

determining, for each candidate transformation operation of the plurality of candidate transformation operations, a respective value for each of one or more third hyperparameters, wherein:

each third hyperparameter corresponds to a different particular position in the sequences of transformation operations selected for transforming training inputs; and

the third hyperparameter corresponding to a particular position in the sequences of transformation operations specifies a probability that the candidate transformation operation is selected as the transformation operation at the particular position in the sequence of transformation operations selected for transforming a particular input.

11. The method of claim 10 , wherein each candidate transformation operation is selected with a same probability in each position in the sequences of transformation operations selected for transforming training inputs.

12. The method of claim 10 , wherein, for each candidate transformation operation, the candidate transformation operation is selected with a same probability in each position in the sequences of transformation operations selected for transforming training inputs.

13. The method of claim 10 , wherein:

the third hyperparameters are machine learned and comprise a respective selection parameter for each candidate transformation operation for each position in the sequences of transformation operations selected for transforming training inputs, and

training a machine learning model using a particular data augmentation policy comprises, for a given training input:

for each position in the sequence of transformation operations selected for transforming the given training input:

processing the respective selection parameter of the position corresponding to each candidate transformation operation using a softmax layer to generate a respective selection probability for each candidate transformation operation, and

selecting a transformation operation for the position in accordance with the respective selection probabilities;

processing the given training input to generate a transformed training input in accordance with the selected sequence of transformation operations;

processing the transformed training input using the machine learning model to generate a training output; and

adjusting current values of the selection parameters based on an error in the training output.

14. The method of claim 1 , wherein determining a quality measure of a machine learning model that has been trained using a data augmentation policy comprises:

determining a performance measure of the machine learning model on the particular machine learning task using validation data comprising a plurality of training inputs;

determining the quality measure based on the performance measure.

15. The method of claim 1 , wherein selecting the final data augmentation policy based on the quality measures of the machine learning models comprises selecting the determined data augmentation policy having the highest quality measure.

16. The method of claim 1 , wherein the training inputs are images and the particular machine learning task is an image processing task comprising classification or regression.

17. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

obtaining a training data set for training a machine learning model having a plurality of network parameters to perform a particular machine learning task, the training data set comprising a plurality of training inputs;

determining a plurality of data augmentation policies, wherein each data augmentation policy defines a procedure for processing a training input to generate a transformed training input, wherein the procedure comprises selecting a sequence of one or more transformation operations to be applied to the training input, and wherein the determining comprises, for each data augmentation policy:

determining a value for a first hyperparameter that specifies a same number of transformation operations in each sequence of transformation operations selected for transforming a respective training input, and

determining a respective value for each of one or more second hyperparameters, wherein the second hyperparameters specify a same magnitude schedule for determining a magnitude for each transformation operation in each sequence of transformation operations selected for transforming a respective training input;

for each data augmentation policy, training the machine learning model using the data augmentation policy, comprising:

determining an augmented training data set by processing the training inputs in the training data set to generate transformed training inputs in accordance with the data augmentation policy,

processing a particular transformed training input in the augmented training data set using current values for the plurality of network parameters of the machine learning model to generate a training output, and

generating a parameter update for the plurality of parameters of the machine learning model based on an error in the training output;

determining, for each data augmentation policy, a quality measure of the machine learning model that has been trained using the data augmentation policy, wherein the quality measure of the machine learning model represents a performance of the machine learning model on the particular machine learning task as a result of training the machine learning model using the data augmentation policy; and

selecting a final data augmentation policy based using the quality measures of the machine learning models.

18. The system of claim 17 , wherein each transformation operation in each sequence of transformation operations is selected from a plurality of candidate transformation operations, and wherein determining a plurality of data augmentation policies further comprises, for each data augmentation policy:

determining, for each candidate transformation operation of the plurality of candidate transformation operations, a respective value for each of one or more third hyperparameters, wherein:

each third hyperparameter corresponds to a different particular position in the sequences of transformation operations selected for transforming training inputs; and

the third hyperparameter corresponding to a particular position in the sequences of transformation operations specifies a probability that the candidate transformation operation is selected as the transformation operation at the particular position in the sequence of transformation operations selected for transforming a particular input.

19. One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations comprising:

obtaining a training data set for training a machine learning model having a plurality of network parameters to perform a particular machine learning task, the training data set comprising a plurality of training inputs;

determining a plurality of data augmentation policies, wherein each data augmentation policy defines a procedure for processing a training input to generate a transformed training input, wherein the procedure comprises selecting a sequence of one or more transformation operations to be applied to the training input, and wherein the determining comprises, for each data augmentation policy:

determining a value for a first hyperparameter that specifies a same number of transformation operations in each sequence of transformation operations selected for transforming a respective training input, and

determining a respective value for each of one or more second hyperparameters, wherein the second hyperparameters specify a same magnitude schedule for determining a magnitude for each transformation operation in each sequence of transformation operations selected for transforming a respective training input;

for each data augmentation policy, training the machine learning model using the data augmentation policy, comprising:

determining an augmented training data set by processing the training inputs in the training data set to generate transformed training inputs in accordance with the data augmentation policy,

processing a particular transformed training input in the augmented training data set using current values for the plurality of network parameters of the machine learning model to generate a training output, and

generating a parameter update for the plurality of parameters of the machine learning model based on an error in the training output;

determining, for each data augmentation policy, a quality measure of the machine learning model that has been trained using the data augmentation policy, wherein the quality measure of the machine learning model represents a performance of the machine learning model on the particular machine learning task as a result of training the machine learning model using the data augmentation policy; and

selecting a final data augmentation policy based using the quality measures of the machine learning models.

20. The non-transitory computer storage media of claim 19 , wherein each transformation operation in each sequence of transformation operations is selected from a plurality of candidate transformation operations, and wherein determining a plurality of data augmentation policies further comprises, for each data augmentation policy:

determining, for each candidate transformation operation of the plurality of candidate transformation operations, a respective value for each of one or more third hyperparameters, wherein:

each third hyperparameter corresponds to a different particular position in the sequences of transformation operations selected for transforming training inputs; and

the third hyperparameter corresponding to a particular position in the sequences of transformation operations specifies a probability that the candidate transformation operation is selected as the transformation operation at the particular position in the sequence of transformation operations selected for transforming a particular input.