LEARNING RATE SCHEDULE FOR TRAINING MACHINE LEARNING BASED LANGUAGE MODELS

1 . A method of training a language model, the method comprising:

receiving a training data set for training the language model;

determining a number of iterations for which the language model should be trained;

initializing parameters of the language model;

repeatedly training the language model for a set of iterations, wherein a learning rate of training is a piecewise function, the training comprising:

training the language model for a first set of iterations for which the learning rate is linearly increasing in value with the number of iterations;

training the language model for a second set of iterations after the first set of iterations, for which the learning rate comprises:

a first term that varies as inverse square root of the number of iterations, and

a second term that has a constant value with respect to the number of iterations; and

training the language model for a third set of iterations after the second set of iterations for which the learning rate is linearly decreasing in value with the number of iterations;

evaluating the trained language model; and

determining based on the evaluation, whether the trained language model should be deployed.

2 . The method of claim 1 , further comprising:

determining based on the evaluation, that the trained language model is ready to be deployed; and

deploying the trained language model.

3 . The method of claim 1 , wherein the third set of iterations has greater than 10% and less than 40% of a total number of iterations.

4 . The method of claim 1 , further comprising:

determining based on the evaluation, that the trained language model needs further training.

5 . The method of claim 4 , wherein the number of iterations is a first number of iterations, the method further comprising:

determining a second number of iterations for which the language model should be further trained; and

repeatedly training the language model for a second set of iterations, wherein the learning rate of training is a piecewise function comprising:

a fourth set of iterations after the third set of iterations for which the learning rate comprises:

the first term that varies as inverse square root of the number of iterations, and

the second term that has the constant value with respect to the number of iterations; and

a fifth set of iterations for which the learning rate is linearly decreasing in value with the number of iterations.

6 . The method of claim 5 , further comprising:

evaluating the language model; and

determining based on the evaluation, whether the trained language model should be deployed.

7 . The method of claim 1 , further comprising, repeating one or more times:

determining an additional number of iterations for which the language model should be further trained;

training the language model for the additional number of iterations; and

evaluating the language model.

8 . The method of claim 1 , further comprising repeating one or more times:

determining an additional number of iterations for which the language model should be further trained;

training the language model for the additional number of iterations, wherein the learning rate of training is a piecewise function comprising:

a fourth set of iterations after the third set of iterations for which the learning rate comprises:

the first term that varies as inverse square root of the number of iterations, and

the second term that has the constant value with respect to the number of iterations; and

a fifth set of iterations for which the learning rate is linearly decreasing in value with the number of iterations; and

evaluating the language model.

9 . A non-transitory computer readable storage medium comprising stored instructions that when executed by one or more computer processors cause the one or more computer processors to:

receive a training data set for training a language model;

determine a number of iterations for which the language model should be trained;

initialize parameters of the language model;

repeatedly train the language model for a set of iterations, wherein a learning rate of training is a piecewise function, the training comprising:

train the language model for a first set of iterations for which the learning rate is linearly increasing in value with the number of iterations;

train the language model for a second set of iterations after the first set of iterations, for which the learning rate comprises:

a first term that varies as inverse square root of the number of iterations, and

a second term that has a constant value with respect to the number of iterations; and

train the language model for a third set of iterations after the second set of iterations for which the learning rate is linearly decreasing in value with the number of iterations;

evaluate the trained language model; and

determine based on the evaluation, whether the trained language model should be deployed.

10 . The non-transitory computer readable storage medium of claim 9 , wherein the stored instructions further cause the one or more computer processors to repeatedly:

determine based on the evaluation, that the trained language model is ready to be deployed; and

deploy the trained language model.

11 . The non-transitory computer readable storage medium of claim 9 , wherein the third set of iterations has greater than 10% and less than 40% of a total number of iterations.

12 . The non-transitory computer readable storage medium of claim 9 , wherein the stored instructions further cause the one or more computer processors to repeatedly:

determine based on the evaluation, that the trained language model needs further training.

13 . The non-transitory computer readable storage medium of claim 12 , wherein the number of iterations is a first number of iterations, wherein the stored instructions further cause the one or more computer processors to:

determine a second number of iterations for which the language model should be further trained; and

repeatedly train the language model for a second set of iterations, wherein the learning rate of training is a piecewise function comprising:

a fourth set of iterations after the third set of iterations for which the learning rate comprises:

the first term that varies as inverse square root of the number of iterations, and

the second term that has the constant value with respect to the number of iterations; and

a fifth set of iterations for which the learning rate is linearly decreasing in value with the number of iterations.

14 . The non-transitory computer readable storage medium of claim 13 , wherein the stored instructions further cause the one or more computer processors to:

evaluate the trained language model; and

determine based on the evaluation, whether the trained language model should be deployed.

15 . The non-transitory computer readable storage medium of claim 9 , wherein the stored instructions further cause the one or more computer processors to repeatedly:

determine an additional number of iterations for which the language model should be further trained;

train the language model for the additional number of iterations; and

evaluate the language model.

16 . The non-transitory computer readable storage medium of claim 9 , wherein the stored instructions further cause the one or more computer processors to repeatedly:

determine an additional number of iterations for which the language model should be further trained; and

train the language model for the additional number of iterations, wherein the learning rate of training is a piecewise function comprising:

a fourth set of iterations after the third set of iterations for which the learning rate comprises:

the first term that varies as inverse square root of the number of iterations, and

the second term that has the constant value with respect to the number of iterations; and

a fifth set of iterations for which the learning rate is linearly decreasing in value with the number of iterations; and

evaluate the language model.

17 . A computer system comprising:

one or more computer processors; and

a non-transitory computer readable storage medium comprising stored program code, the stored program code comprising instructions, the stored instructions when executed cause the one or more computer processors to:

receive a training data set for training a language model;

determine a number of iterations for which the language model should be trained;

initialize parameters of the language model;

repeatedly train the language model for a set of iterations, wherein a learning rate of training is a piecewise function, the training comprising:

train the language model for a first set of iterations for which the learning rate is linearly increasing in value with the number of iterations;

train the language model for a second set of iterations after the first set of iterations, for which the learning rate comprises:

a first term that varies as inverse square root of the number of iterations, and

a second term that has a constant value with respect to the number of iterations; and

train the language model for a third set of iterations after the second set of iterations for which the learning rate is linearly decreasing in value with the number of iterations;

evaluate the trained language model; and

determine based on the evaluation, whether the trained language model should be deployed.

18 . The computer system of claim 17 , wherein the number of iterations is a first number of iterations, wherein the stored instructions further cause the one or more computer processors to:

determine based on the evaluation, that the trained language model needs further training;

determine a second number of iterations for which the language model should be further trained;

repeatedly train the language model for a second set of iterations, wherein the learning rate of training is a piecewise function comprising:

a fourth set of iterations after the third set of iterations for which the learning rate comprises:

the first term that varies as inverse square root of the number of iterations, and

the second term that has the constant value with respect to the number of iterations; and

a fifth set of iterations for which the learning rate is linearly decreasing in value with the number of iterations.

19 . The computer system of claim 17 , wherein the stored instructions further cause the one or more computer processors to repeatedly:

determine an additional number of iterations for which the language model should be further trained;

train the language model for the additional number of iterations; and

evaluate the language model.

20 . The computer system of claim 17 , wherein the stored instructions further cause the one or more computer processors to repeatedly:

determining an additional number of iterations for which the language model should be further trained;

training the language model for the additional number of iterations, wherein the learning rate of training is a piecewise function comprising:

a fourth set of iterations after the third set of iterations for which the learning rate comprises:

the first term that varies as inverse square root of the number of iterations, and

the second term that has the constant value with respect to the number of iterations; and

a fifth set of iterations for which the learning rate is linearly decreasing in value with the number of iterations; and

evaluating the language model.