In-flight scaling of machine learning training jobs
Techniques for in-flight scaling of machine learning training jobs are described. A request to execute a machine learning (ML) training job is received within a provider network, and the ML training job is executed using a first one or more compute instances. Upon a determination that a performance characteristic of the ML training job satisfies a scaling condition, a second one or more compute instances are added to the ML training job while the first one or more compute instances continue to execute portions of the ML training job.
1. A computer-implemented method comprising:
executing code on a computing device including a processor to implement a model training system in a provider network;
executing code on another computing device including a processor to implement a training control system in the provider network;
executing a machine learning (ML) training job using a first one or more compute instances of the model training system, wherein each of the first one or more compute instances performs a plurality of iterations of a work routine for the ML training job, each iteration including obtaining an identifier of a work unit from a progress manager of the training control system, obtaining a current state associated with the ML training job from a parameter server of the training control system, executing logic to update the current state associated with the ML training job after completing the work unit, sending the updated current state associated with the ML training job to the parameter server of the training control system, and sending a message to the progress manager of the training control system indicating that the work unit is complete;
determining, by the progress manager of the training control system, that a first compute instance from among the first one or more compute instances of the model training system obtained a first identifier of a first work unit for the ML training job from the progress manager of the training control system at a first time and updated the current state of the first work unit for the ML training job at a second time;
deriving, by the progress manager of the training control system from the first time and the second time, a speed of the ML training job;
determining, by the progress manager of the training control system by comparing a predicted training time to a threshold and based at least in part on the deriving of the speed of the ML training job, that a progress of the ML training job is not satisfactory;
adding, by the model training system based at least in part on the determining that the progress of the ML training job is not satisfactory, a second one or more compute instances of the model training system to the ML training job while the first one or more compute instances of the model training system continue to execute portions of the ML training job, wherein the second one or more compute instances are of a different type than the first one or more compute instances, and wherein the different type is selected based on an execution characteristic of the ML training job that is determined to be problematic hindering performance of the ML training job;
providing, to the second one or more compute instances of the model training system, an identifier of the parameter server and an identifier of a set of data that the second one or more compute instances of the model training system is to process;
determining, by the progress manager of the training control system, that the progress of the ML training job is satisfactory; and
removing, by the model training system based at least in part on the determining that the progress of the ML training job is satisfactory, one or more compute instances of the second one or more compute instances of the model training system or of the first one or more compute instances of the model training system from the ML training job.
2. The computer-implemented method of claim 1 , wherein the determining that the progress of the ML training job is satisfactory comprises comparing the predicted training time to the threshold or comparing a learning rate to another threshold.
3. The computer-implemented method of claim 1 , the different type of the second one or more compute instances comprising at least one of:
a different number of virtual processors;
a different amount of random access memory (RAM);
a different amount of nonvolatile storage;
a different amount of networking capacity; or
a different type or amount of graphical processing units (GPUs) or field programmable gate arrays (FPGAs) available.
4. The computer-implemented method of claim 1 , wherein training code is executed within a container.
5. A computer-implemented method comprising:
executing code on a computing device including a processor to implement a model training system in a provider network;
executing code on another computing device including a processor to implement a training control system in the provider network;
executing, using a first one or more compute instances of the model training system, a machine learning (ML) training job by:
obtaining, from a progress manager of the training control system, a first identifier of a first work unit the first one or more compute instances are to perform for the ML training job;
obtaining, from a parameter server of the training control system, job state data associated with the ML training job;
after completing the first work unit, sending a message to the parameter server of the training control system to update the job state data associated with the ML training job; and
after completing the first work unit, sending a message to the progress manager of the training control system indicating that the first work unit has been completed;
determining, by the progress manager of the training control system, that the first one or more compute instances of the model training system obtained the first identifier of the first work unit at a first time and the job state data was updated at a second time;
deriving, by the progress manager of the training control system, from the first time and the second time, a speed of the ML training job;
determining, by the progress manager of the training control system, by comparing a predicted training time to a threshold and based at least in part on the deriving of the speed of the ML training job, that a progress of the ML training job is not satisfactory;
adding, by the model training system, based at least in part on the determining that the progress of the ML training job is not satisfactory, a second one or more compute instances of the model training system to the ML training job while the first one or more compute instances of the model training system continue to execute portions of the ML training job, wherein the second one or more compute instances are of a different type than the first one or more compute instances, and wherein the different type is selected based on an execution characteristic of the ML training job that is determined to be problematic hindering performance of the ML training job;
providing, to the second one or more compute instances of the model training system, an identifier of the parameter server and an identifier of a set of data that the second one or more compute instances of the model training system is to process;
determining, by the progress manager of the training control system, that the progress of the ML training job is satisfactory; and
removing, by the model training system, based at least in part on the determining that the progress of the ML training job is satisfactory, one or more compute instances of the second one or more compute instances of the model training system or the first one or more compute instances of the model training system from the ML training job.
6. The computer-implemented method of claim 5 , wherein the determining that the progress of the ML training job is satisfactory comprises comparing the predicted training time to the threshold or comparing a learning rate to another threshold.
7. The computer-implemented method of claim 5 , wherein the determining that a progress of the ML training job is not satisfactory further comprises comparing a learning rate to another threshold.
8. The computer-implemented method of claim 5 , the different type of the second one or more compute instances comprising at least one of:
a different number of virtual processors;
a different amount of random access memory (RAM);
a different amount of nonvolatile storage;
a different amount of networking capacity; or
a different type or amount of graphical processing units (GPUs) or field programmable gate arrays (FPGAs) available.
9. The computer-implemented method of claim 5 , wherein training code is executed within a container.
10. The computer-implemented method of claim 5 , wherein executing the ML training job by the first one or more compute instances further includes obtaining, from a job status manager, an identifier of a parameter server.
11. The computer-implemented method of claim 5 , further comprising:
receiving a request from a client to remove one or more compute instances from the ML training job, wherein the one or more compute instances are of the second one or more compute instances or of the first one or more compute instances; and
removing the one or more compute instances from the ML training job.
12. The computer-implemented method of claim 5 , further comprising:
receiving a request from a client to add one or more compute instances to the ML training job; and
adding the one or more compute instances to the ML training job while the first one or more compute instances and the second one or more compute instances continue to execute portions of the ML training job.
13. A system comprising:
a model training system implemented by a first one or more electronic devices; and
a training control system implemented by a second one or more electronic devices, the training control system including second instructions that upon execution cause the training control system to:
cause the model training system to execute a machine learning (ML) training job using a first one or more compute instances of the model training system such that the model training system:
obtains, from a progress manager of the training control system, a first identifier of a first work unit the first one or more compute instances are to perform for the ML training job;
obtains, from a parameter server of the training control system, job state data associated with the ML training job;
after completing the first work unit, sends a message to the parameter server of the training control system to update the job state data associated with the ML training job; and
after completing the first work unit, sends a message to the progress manager of the training control system indicating that the first work unit has been completed;
determine, by the progress manager of the training control system, that the first one or more compute instances of the model training system obtained the first identifier of the first work unit at a first time and the job state data was updated at a second time;
derive, by the progress manager of the training control system, from the first time and the second time, a speed of the ML training job;
determine, by the progress manager of the training control system, by comparing a predicted training time to a threshold and based at least in part on the deriving of the speed of the ML training job, that a progress of the ML training job is not satisfactory;
cause, by the model training system, based at least in part on the determining that the progress of the ML training job is not satisfactory, a second one or more compute instances of the model training system to be added to the ML training job while the first one or more compute instances of the model training system continue to execute portions of the ML training job, wherein the second one or more compute instances are of a different type than the first one or more compute instances, and wherein the different type is selected based on an execution characteristic of the ML training job that is determined to be problematic hindering performance of the ML training job;
provide, to the second one or more compute instances of the model training system, an identifier of the parameter server and an identifier of a set of data that the second one or more compute instances of the model training system is to process;
determine, by the progress manager of the training control system, that the progress of the ML training job is satisfactory; and
cause, by the model training system, based at least in part on the determining that the progress of the ML training job is satisfactory, one or more compute instances of the second one or more compute instances of the model training system or the first one or more compute instances of the model training system to be removed from the ML training job.
14. The system of claim 13 , wherein to determine that the progress of the ML training job is satisfactory the instructions upon execution further cause the training control system to compare the predicted training time to the threshold or compare a learning rate to another threshold.
15. The system of claim 13 , wherein the determining that a progress of the ML training job is not satisfactory further comprises comparing a learning rate to another threshold.
16. The system of claim 13 , the different type of the second one or more compute instances comprising at least one of:
a different number of virtual processors;
a different amount of random access memory (RAM);
a different amount of nonvolatile storage;
a different amount of networking capacity; or
a different type or amount of graphical processing units (GPUs) or field programmable gate arrays (FPGAs) available.
17. The system of claim 13 , wherein training code is executed within a container.
18. The system of claim 13 , wherein the first compute instance is further to obtain, from a job status manager, an identifier of a parameter server.
19. The system of claim 13 , wherein the second instructions upon execution further cause the training control system to:
upon a receipt of a request from a client to add one or more compute instances to the ML training job, cause the one or more compute instances to be added to the ML training job while the first one or more compute instances and the second one or more compute instances continue to execute portions of the ML training job.
20. The system of claim 13 , wherein the second instructions upon execution further cause the training control system to:
upon a receipt of a request from a client to remove one or more compute instances from the ML training job, wherein the one or more compute instances are of the second one or more compute instances or of the first one or more compute instances, cause the one or more compute instances to be removed to the ML training job.