Dynamic distributed training of machine learning models
Computer hardware and/or software that performs the following operations: (i) updating a machine learning model by synchronously applying, to the machine learning model, a first set of training results received from a set of trainers having respective training datasets; (ii) receiving, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results; and (iii) based, at least in part, on the first set of metrics, determining to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers.
1 . A computer-implemented method comprising:
updating, by one or more processors, a machine learning model by synchronously applying, to the machine learning model, a first set of training results received at a sub-system from a set of trainers having respective training datasets, as synchronous training;
receiving at the sub-system, by one or more processors, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results;
based, at least in part, on the first set of metrics, determining, by one or more processors on a configuration computer communicating with the sub-system, to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers;
in response to the first set of metrics indicating that an amount of time taken by a first trainer of the set of trainers to produce respective training results differs from an amount of time taken by one or more other trainers to produce respective training results by at least a threshold amount of time, switching, using the configuration computer, to an asynchronous training mode, the switching to the asynchronous training mode being based on the first trainer being a slow trainer; and
switching back to the synchronous training, using the configuration computer, when the slow trainer is no longer the slow trainer when all the one or more trainers of the set of trainers report similar training times within a threshold distance.
2 . The computer-implemented method of claim 1 , further comprising:
updating, by one or more processors, the machine learning model by applying, to the machine learning model, at a first time, a second set of training results received from a first trainer of the set of trainers; and
updating, by one or more processors, the machine learning model by applying, to the machine learning model, at a second time, a third set of training results received from a second trainer of the set of trainers.
3 . The computer-implemented method of claim 2 , further comprising:
generating, by one or more processors, a first updated set of machine learning model parameters based, at least in part, on the synchronous application, to the machine learning model, of the first set of training results; and
communicating, by one or more processors, the first updated set of machine learning model parameters to the set of trainers,
wherein the second set of training results and the third set of training results are based, at least in part, on the first updated set of machine learning model parameters.
4 . The computer-implemented method of claim 1 , further comprising:
receiving, by one or more processors, from one or more trainers of the set of trainers, a second set of metrics pertaining to at least some training results of a second set of training results; and
based, at least in part, on the second set of metrics, determining, by one or more processors, to subsequently update the machine learning model via synchronous application of subsequent training results received from the respective trainers of the set of trainers.
5 . The computer-implemented method of claim 4 , further comprising:
receiving, by one or more processors, respective training results from each trainer of the set of trainers;
combining, by one or more processors, the respective training results from each trainer of the set of trainers into a third set of training results; and
updating, by one or more processors, the machine learning model by synchronously applying, to the machine learning model, the third set of training results.
6 . The computer-implemented method of claim 4 , wherein the first set of metrics indicates that an amount of time taken by a first trainer of the set of trainers to produce respective training results of the first set of training results differs from an amount of time taken by one or more other trainers of the set of trainers to produce respective training results of the first set of training results by at least a threshold amount of time.
7 . The computer-implemented method of claim 6 , wherein the second set of metrics indicates that an amount of time taken by the first trainer to produce respective training results of the second set of training results differs from an amount of time taken by the one or more other trainers to produce respective training results of the second set of training results by less than the threshold amount of time.
8 . The computer-implemented method of claim 6 , wherein the second set of metrics indicates that an amount of difference between a current measure of machine learning model convergence and a desired measure of machine learning model convergence exceeds a threshold amount of difference.
9 . The computer-implemented method of claim 1 , further comprising:
configuring, by one or more processors, the set of trainers for training by sending an initial copy of the machine learning model to each trainer of the set of trainers.
10 . The computer-implemented method of claim 1 , wherein the first set of training results includes respective machine learning model gradients for each trainer of the set of trainers.
11 . A computer program product comprising one or more computer readable storage media and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by one or more processors to cause the one or more processors to perform a method comprising:
updating a machine learning model by synchronously applying, to the machine learning model, a first set of training results received at a sub-system from a set of trainers having respective training datasets, as synchronous training;
receiving, at the sub-system, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results;
based, at least in part, on the first set of metrics, determining on a configuration computer communicating with the sub-system to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers;
in response to the first set of metrics indicating that an amount of time taken by a first trainer of the set of trainers to produce respective training results differs from an amount of time taken by one or more other trainers to produce respective training results by at least a threshold amount of time, switching, using the configuration computer, to an asynchronous training mode, the switching to the asynchronous training mode being based on the first trainer being a slow trainer; and
switching back to the synchronous training, using the configuration computer, when the slow trainer is no longer the slow trainer when all the one or more trainers of the set of trainers report similar training times within a threshold distance.
12 . The computer program product of claim 11 , the method further comprising:
updating the machine learning model by applying, to the machine learning model, at a first time, a second set of training results received from a first trainer of the set of trainers; and
updating the machine learning model by applying, to the machine learning model, at a second time, a third set of training results received from a second trainer of the set of trainers.
13 . The computer program product of claim 12 , the method further comprising:
generating a first updated set of machine learning model parameters based, at least in part, on the synchronous application, to the machine learning model, of the first set of training results; and
communicating the first updated set of machine learning model parameters to the set of trainers,
wherein the second set of training results and the third set of training results are based, at least in part, on the first updated set of machine learning model parameters.
14 . The computer program product of claim 11 , the method further comprising:
receiving, from one or more trainers of the set of trainers, a second set of metrics pertaining to at least some training results of a second set of training results; and
based, at least in part, on the second set of metrics, determining to subsequently update the machine learning model via synchronous application of subsequent training results received from the respective trainers of the set of trainers.
15 . The computer program product of claim 14 , the method further comprising:
receiving respective training results from each trainer of the set of trainers;
combining the respective training results from each trainer of the set of trainers into a third set of training results; and
updating the machine learning model by synchronously applying, to the machine learning model, the third set of training results.
16 . A computer system comprising:
one or more processors; and
one or more computer readable storage media;
wherein:
the one are more processors are structured, located, connected and/or programmed to execute program instructions collectively stored on the one or more computer readable storage media; and
the program instructions, when executed by the one or more processors, cause the one or more processors to perform a method comprising:
updating a machine learning model by synchronously applying, to the machine learning model, a first set of training results received at a sub-system from a set of trainers having respective training datasets, as synchronous training;
receiving at the sub-system, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results;
based, at least in part, on the first set of metrics, determining on a configuration computer communicating with the sub-system, to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers;
in response to the first set of metrics indicating that an amount of time taken by a first trainer of the set of trainers to produce respective training results differs from an amount of time taken by one or more other trainers to produce respective training results by at least a threshold amount of time, switching, using the configuration computer, to an asynchronous training mode, the switching to the asynchronous training mode being based on the first trainer being a slow trainer; and
switching back to the synchronous training, using the configuration computer, when the slow trainer is no longer the slow trainer when all the one or more trainers of the set of trainers report similar training times within a threshold distance.
17 . The computer system of claim 16 , the method further comprising:
updating the machine learning model by applying, to the machine learning model, at a first time, a second set of training results received from a first trainer of the set of trainers; and
updating the machine learning model by applying, to the machine learning model, at a second time, a third set of training results received from a second trainer of the set of trainers.
18 . The computer system of claim 17 , the method further comprising:
generating a first updated set of machine learning model parameters based, at least in part, on the synchronous application, to the machine learning model, of the first set of training results; and
communicating the first updated set of machine learning model parameters to the set of trainers,
wherein the second set of training results and the third set of training results are based, at least in part, on the first updated set of machine learning model parameters.
19 . The computer system of claim 16 , the method further comprising:
receiving, from one or more trainers of the set of trainers, a second set of metrics pertaining to at least some training results of a second set of training results; and
based, at least in part, on the second set of metrics, determining to subsequently update the machine learning model via synchronous application of subsequent training results received from the respective trainers of the set of trainers.
20 . The computer system of claim 19 , the method further comprising:
receiving respective training results from each trainer of the set of trainers;
combining the respective training results from each trainer of the set of trainers into a third set of training results; and
updating the machine learning model by synchronously applying, to the machine learning model, the third set of training results.