Peer assisted distributed architecture for training machine learning models

1. A method for distributing training of a machine learning model across a plurality of computing devices, comprising:

receiving, from a computing device of the plurality of computing devices, a request for a set of outstanding jobs for training part of a machine learning model;

transmitting, to the computing device, information identifying the set of outstanding jobs;

receiving, from the computing device, a selected job for execution on the computing device from the set of outstanding jobs, the selected job being associated with a machine learning model to be trained;

transmitting, to the computing device, a chunk of training data associated with the selected job and one or more parameters associated with the selected job; and

taking one or more actions with respect to the chunk of data associated with the selected job based on a response from the computing device.

2. The method of claim 1 , wherein the taking one or more actions comprises:

receiving, from the computing device, a set of weights calculated from the chunk of training data associated with the selected job; and

adding the received set of weights to the machine learning model associated with the selected job.

3. The method of claim 1 , wherein the one or more parameters include a deadline for reporting a result of processing the chunk of data associated with the selected job, and the taking one or more actions comprises:

determining, that the computing device did not generate a valid response before the deadline; and

resetting a status for the chunk of data associated with the selected job such that the chunk of data associated with the selected job can be assigned to another computing device in the plurality of computing devices.

4. The method of claim 3 , further comprising:

adjusting a performance metric associated with the computing device.

5. The method of claim 4 , further comprising:

receiving, from the computing device, a subsequent request for a set of outstanding jobs to execute on the computing device;

selecting, from the outstanding jobs, a group of outstanding jobs having performance metrics less than or equal to the adjusted performance metric associated with the computing device; and

transmitting, to the computing device, the selected group of outstanding jobs for selection.

6. The method of claim 1 , further comprising:

periodically moving chunks of training data including the chunk of training data across a plurality of training data repositories.

7. The method of claim 6 , wherein the chunk of training data is selected based on a geographic proximity of a data repository storing the chunk of training data to the computing device.

8. The method of claim 1 , wherein the chunk of training data is selected based on information identifying a position within a set of training data for the selected job corresponding to chunks of training data that have been previously dispatched for processing by the plurality of computing devices.

9. The method of claim 1 , wherein a size of the chunk of training data is selected based on one or more performance characteristics of the computing device.

10. The method of claim 9 , further comprising:

determining the performance characteristics of the computing device are determined by:

transmitting, for execution on the computing device, a benchmarking suite, and

receiving a score from the benchmarking suite indicative of a level of performance of the computing device;

wherein ranges of scores are mapped to different sizes of chunks of training data.

11. The method of claim 1 , further comprising:

transmitting, to the computing device, a training algorithm to be used by the computing device in performing the selected job using the chunk of data and the one or more parameters.

12. A system, comprising:

a processor; and

a memory having instructions stored thereon which, when executed by the processor, performs an operation for distributing training of a machine learning model across a plurality of computing devices, the operation comprising:

receiving, from a computing device of the plurality of computing devices, a request for a set of outstanding jobs for training part of a machine learning model;

transmitting, to the computing device, information identifying the set of outstanding jobs;

receiving, from the computing device, a selected job for execution on the computing device from the set of outstanding jobs, the selected job being associated with a machine learning model to be trained;

transmitting, to the computing device, a chunk of training data associated with the selected job and one or more parameters associated with the selected job; and

taking one or more actions with respect to the chunk of data associated with the selected job based on a response from the computing device.

13. The system of claim 12 , wherein the taking one or more actions comprises:

receiving, from the computing device, a set of weights calculated from the chunk of training data associated with the selected job; and

adding the received set of weights to the machine learning model associated with the selected job.

14. The system of claim 12 , wherein the one or more parameters include a deadline for reporting a result of processing the chunk of data associated with the selected job, and the taking one or more actions comprises:

determining, that the computing device did not generate a valid response before the deadline; and

resetting a status for the chunk of data associated with the selected job such that the chunk of data associated with the selected job can be assigned to another computing device in the plurality of computing devices.

15. The system of claim 14 , further comprising:

adjusting a performance metric associated with the computing device.

16. The system of claim 15 , further comprising:

receiving, from the computing device, a subsequent request for a set of outstanding jobs to execute on the computing device;

selecting, from the outstanding jobs, a group of outstanding jobs having performance metrics less than or equal to the adjusted performance metric associated with the computing device; and

transmitting, to the computing device, the selected group of outstanding jobs for selection.

17. The system of claim 11 , further comprising:

periodically moving chunks of training data including the chunk of training data across a plurality of training data repositories, wherein the chunk of training data is selected based on a geographic proximity of a data repository storing the chunk of training data to the computing device.

18. The system of claim 11 , wherein the chunk of training data is selected based on information identifying a position within a set of training data for the selected job corresponding to chunks of training data that have been previously dispatched for processing by the plurality of computing devices.

19. The system of claim 11 , wherein a size of the chunk of training data is selected based on one or more performance characteristics of the computing device.

20. A computer-readable medium having instructions stored thereon which, when executed by a processor, performs an operation for distributing training of a machine learning model across a plurality of computing devices, the operation comprising:

receiving, from a computing device of the plurality of computing devices, a request for a set of outstanding jobs for training part of a machine learning model;

transmitting, to the computing device, information identifying the set of outstanding jobs;

receiving, from the computing device, a selected job for execution on the computing device from the set of outstanding jobs, the selected job being associated with a machine learning model to be trained;

transmitting, to the computing device, a chunk of training data associated with the selected job and one or more parameters associated with the selected job; and

taking one or more actions with respect to the chunk of data associated with the selected job based on a response from the computing device.