Machine learning model training checkpoints

1 . A method, comprising:

initiating a first training operation for training a machine learning model;

at a first predetermined checkpoint during the first training operation, generating first checkpoint information comprising a representation of the machine learning model in a first partially trained state and feature vectors generated from training data used in the first training operation of the machine learning model;

at a second dynamic checkpoint during the first training operation, generating second checkpoint information comprising a representation of the machine learning model in a second partially trained state, wherein the second dynamic checkpoint is based at least in part on a change in a loss function measured after a most recent training iteration and is generated upon determining, after the most recent training iteration, that a metric for the change in the loss function is satisfied;

storing the first checkpoint information and storing the second checkpoint information in a non-volatile storage medium; and

continuing the first training operation for training the machine learning model after the first predetermined checkpoint and after the second dynamic checkpoint.

2 . The method of claim 1 , wherein the first training operation comprises a first number of training iterations, and

wherein the first predetermined checkpoint is set at a point after a second number of training iterations has been completed, wherein the second number of training iterations is less than the first number of training iterations.

3 . The method of claim 2 , further comprising:

pausing or cancelling the first training operation;

loading the first checkpoint information or the second checkpoint information from the non-volatile storage medium; and

initiating a second training operation for training the machine learning model starting in the first partially trained state or the second partially trained state using the stored first checkpoint information or the stored second checkpoint information, respectively, respectfully, the second training operation comprising additional training iterations.

4 . The method of claim 3 , further comprising:

at a third predetermined checkpoint during the second training operation, generating third checkpoint information comprising a representation of the machine learning model in a third partially trained state, wherein the additional training iterations of the second training operation correspond to training iterations remaining from the first number of training iterations after the second number of training iterations are completed;

storing the third checkpoint information in the non-volatile storage medium; and

continuing the second training operation for training the machine learning model after the third predetermined checkpoint.

5 . The method of claim 4 , wherein the first training operation for training the machine learning model uses a first set of training data and the second training operation for training the machine learning model uses a second set of training data different from the first set of training data.

6 . The method of claim 5 , wherein the first checkpoint information further comprises loss values for the first set of training data determined in a last training iteration before the first predetermined checkpoint and the third checkpoint information further comprises loss values for the second set of training data determined in a last training iteration before the third predetermined checkpoint.

7 . The method of claim 1 , further comprising:

generating and storing checkpoint information at a plurality of additional predetermined or dynamic checkpoints during the first training operation;

completing the first training operation;

performing an evaluation of a trained machine learning model generated by the completion of the first training operation;

obtaining, based on the evaluation, the checkpoint information generated at a last one of the plurality of additional predetermined or dynamic checkpoints; and

initiating a second training operation for training the machine learning model starting in a partially trained state represented in the checkpoint information generated at the last one of the plurality of additional predetermined or dynamic checkpoints.

8 . The method of claim 7 , wherein completing the first training operation comprises completing a predetermined number of training iterations or achieving a predetermined accuracy.

9 . The method of claim 8 , wherein performing the evaluation of the trained machine learning model comprises providing evaluation data as input to the trained machine learning model to obtain an evaluation output, and comparing the evaluation output to an evaluation criteria.

10 . The method of claim 1 , further comprising:

generating an operational machine learning model based on the first checkpoint information or the second checkpoint information;

passing a user-selected set of data through the operational machine learning model to generate output data; and

storing the output data in the non-volatile storage medium.

11 . The method of claim 1 , wherein the first checkpoint information comprises a matrix representation of the machine learning model in the first partially trained state, and a first tensor comprising weight values of the machine learning model in the first partially trained state, and wherein the second checkpoint information comprises a matrix representation of the machine learning model in the second partially trained state, feature vectors generated from training data used in the first training operation of the machine learning model, and a second tensor comprising weight values of the machine learning model in the second partially trained state.

12 . A system, comprising:

a processor; and

a memory device containing instructions which, when executed by the processor, cause the processor to:

store, in a non-volatile storage medium, first checkpoint information comprising:

a representation of a machine learning model in a first partially trained state corresponding to a first predetermined checkpoint during a first training operation for the machine learning model,

and feature vectors generated from training data used in the first training operation of the machine learning model;

store, in the non-volatile storage medium, second checkpoint information comprising:

a representation of the machine learning model in a second partially trained state corresponding to a second dynamic checkpoint during the first training operation for the machine learning model, wherein the second dynamic checkpoint is based at least in part on a change in a loss function measured after a most recent training iteration of the first training operation and is generated upon determining, after the most recent training iteration, that a metric for the change in the loss function is satisfied;

load the first checkpoint information or the second checkpoint information from the non-volatile storage medium; and

initiate a second training operation for training the machine learning model starting in the first partially trained state or the second partially trained state using the loaded first checkpoint information or loaded second checkpoint information, respectively.

13 . The system of claim 12 , wherein the instructions, when executed by the processor, further cause the processor to perform the first training operation, wherein the first training operation comprises a first number of training iterations, and

wherein the first predetermined checkpoint is set at a point after a second number of training iterations has been completed, wherein the second number of training iterations is less than the first number of training iterations.

14 . The system of claim 12 , wherein the instructions, when executed by the processor, further cause the processor to:

at a third predetermined checkpoint during the second training operation, generate third checkpoint information comprising a representation of the machine learning model in a third partially trained state;

store the third checkpoint information in the non-volatile storage medium; and

continue the second training operation for training the machine learning model after the third predetermined checkpoint.

15 . The system of claim 14 , wherein the first training operation for training the machine learning model uses a first set of training data and the second training operation for training the machine learning model uses a second set of training data different from the first set of training data.

16 . The system of claim 15 , wherein the first checkpoint information further comprises loss values for the first set of training data determined in a last training iteration before the first predetermined checkpoint and the third checkpoint information further comprises loss values for the second set of training data determined in a last training iteration before the third predetermined checkpoint.

17 . The system of claim 12 , wherein the instructions, when executed by the processor, further cause the processor to:

generate an operational machine learning model based on the first checkpoint information or the second checkpoint information;

pass a user-selected set of data through the operational machine learning model to generate output data; and

store the output data in the non-volatile storage medium.

18 . The system of claim 12 , wherein the first checkpoint information comprises a matrix representation of the machine learning model in the first partially trained state, and a first tensor comprising weight values of the machine learning model in the first partially trained state, and wherein the second checkpoint information comprises a matrix representation of the machine learning model in the second partially trained state, feature vectors generated from training data used in the first training operation of the machine learning model, and a second tensor comprising weight values of the machine learning model in the second partially trained state.

19 . A computer program product comprising code stored in a non-transitory tangible computer-readable storage medium, the code comprising:

code for initiating a first training operation for training a machine learning model;

code for, at a first predetermined checkpoint during the first training operation, generating first checkpoint information comprising a representation of the machine learning model in a first partially trained state and feature vectors generated from training data used in the first training operation of the machine learning model;

code for storing the first checkpoint information in a non-volatile storage medium;

code for continuing the first training operation for training the machine learning model after the first predetermined checkpoint;

code for, at a second dynamic checkpoint during the first training operation, generating second checkpoint information comprising a representation of the machine learning model in a second partially trained state, wherein the second dynamic checkpoint is based at least in part on a change in a loss function measured after a most recent training iteration and is generated upon determining, after the most recent training iteration, that a metric for the change in the loss function is satisfied;

code for storing the second checkpoint information in the non-volatile storage medium; and

code for continuing the first training operation for training the machine learning model after the second dynamic checkpoint.

20 . The computer program product of claim 19 , wherein the computer program product further comprises:

code for pausing or cancelling the first training operation;

code for loading the first checkpoint information or the second checkpoint information from the non-volatile storage medium; and

code for initiating a second training operation for training the machine learning model starting in the first partially trained state using the loaded first checkpoint information or second checkpoint information.

21 . The computer program product of claim 20 , wherein the first training operation for training the machine learning model uses a first set of training data and the second training operation for training the machine learning model uses a second set of training data different from the first set of training data.

22 . The computer program product of claim 19 , wherein the computer program product further comprises:

code for generating an operational machine learning model based on the first checkpoint information or the second checkpoint information;

code for passing a user-selected set of data through the operational machine learning model to generate output data; and

code for storing the output data in the non-volatile storage medium.

23 . The computer program product of claim 19 , wherein the first checkpoint information comprises a matrix representation of the machine learning model in the first partially trained state, and a first tensor comprising weight values of the machine learning model in the first partially trained state, and

wherein the second checkpoint information comprises a matrix representation of the machine learning model in the second partially trained state, feature vectors generated from training data used in the first training operation of the machine learning model, and a second tensor comprising weight values of the machine learning model in the second partially trained state.