Efficient online training for machine learning
An example system includes prediction workers, training workers, and a parameter server. The prediction workers store a local copy of a machine-learned model and run the mode exclusively in serving mode. The training workers store a local copy of a machine-learned model and a local snapshot and run the local copy exclusively in training mode and compare the local model or state to the snapshot after training to send delta updates to the parameter server after training. The parameter server aggregates received delta updates into a master copy of the model, sends the aggregated updates back to training workers and provides two types of updates; a real-time update based on a comparison of the master model with a local snapshot, and a full update. The real-time update occurs at least an order of magnitude more frequently than the full update and includes a subset of the weights in the model.
1. A system comprising:
at least one processor;
a plurality of prediction workers, each prediction worker being executable by the at least one processor and being configured to:
store a respective local copy of a machine-learned model,
receive a request from a client for a prediction, the request including input data,
generate the prediction by running the input data through the local copy of the machine-learned model without updating the local copy, and
provide the prediction for the input data to the client;
a plurality of training workers, each training worker being executable by the at least one processor and being configured to:
store a respective local copy of the machine-learned model or a local copy of a state of the machine-learned model,
store a respective snapshot of the machine-learned model or the state of the machine-learned model,
receive training examples,
train the local copy of the machine-learned model using the training examples, and
periodically send one or more delta updates to a parameter server after training, the delta updates representing differences between the snapshot and the local copy of the machine-learned model or the local copy of the state; and
the parameter server, the parameter server being configured to:
receive the delta updates from the plurality of training workers,
aggregate the delta updates into a master copy of the machine-learned model to generate aggregated updates,
send the aggregated updates back to training workers,
on a first periodic basis provide a real-time update by:
determining a subset of weights for a real-time update, the subset being less than all of the weights for the master copy and the subset including weights having a change that satisfies a threshold, and
providing the subset of weights as a real-time update to each of the plurality of prediction workers, and
on a second periodic basis, provide a full update of the master copy to each of the plurality of prediction workers and to each of the plurality of training workers, wherein the full update is used to replace each respective local copy,
wherein the first periodic basis occurs at least an order of magnitude more frequently than the second periodic basis.
2. The system of claim 1 , wherein the first period is measured in seconds and the second period is measured at least in hours.
3. The system of claim 1 , wherein the change is a rate of change over time.
4. The system of claim 1 , each of the plurality of training workers being further configured to send first metrics to a quality assurance agent, the quality assurance agent being executable by the at least one processor, and the system further comprises:
an observation worker being executable by the at least one processor and being configured to:
store a respective local copy of a machine-learned model;
receive the training examples;
generate a prediction by running the training examples through the local copy of the machine-learned model; and
provide second metrics the quality assurance agent based on the generating,
wherein the parameter server is configured to provide the observation worker with the real-time update and the full update, and
wherein the quality assurance agent provides an alert when a difference between the first metrics and the second metrics satisfies a second threshold.
5. The system of claim 4 , the parameter server is further configured to, responsive to the alert, cease providing the real-time update to each of the plurality of prediction workers.
6. The system of claim 1 , the parameter server being configured to:
receive a request for an initial update from a requesting worker, the requesting worker being one of the plurality of prediction workers;
determine a second set of weights having a change that satisfies a second threshold; and
provide the second set of weights to the requesting worker.
7. The system of claim 6 , wherein the change that satisfies the second threshold is an absolute change.
8. The system of claim 6 , wherein the second set of weights are weights identified as with highest impact on model quality.
9. The system of claim 1 , wherein each respective local copy of the machine-learned model is a subset of the master copy.
10. A method comprising:
receiving training data;
providing the training data to an observation worker and to a training worker of a plurality of workers, the observation worker and the training worker each having a respective local copy of a machine-learned model and each being executable by at least one processor;
receiving, at a parameter server, delta updates from the plurality of training workers, the delta updates being related to changes in weights for the machine-learned model, wherein the parameter server provides, on a first periodic basis, a subset of weights for a real-time update to the observation worker and to a plurality of prediction workers and provides, on a second periodic basis, a full update of weights to the observation worker and to the plurality of prediction workers, wherein the plurality of prediction workers run a local copy of the machine-learned model in a serving mode to provide a response to a input without updating the local copy of the machine-learned model;
generating first model quality metrics for the training data using the observation worker by the observation worker running the training data through the local copy of the machine-learned model in a serving mode without updating the local copy of the machine-learned model;
generating second model quality metrics for the training data from the training worker by the training worker running the training data through the local copy of the machine-learned model in a training mode that updates the local copy of the model; and
sending an alert when a difference between the first model quality metrics and the second model quality metrics satisfies a threshold.
11. The method of claim 10 , wherein the first periodic basis occurs at least an order of magnitude more frequently than the second periodic basis.
12. The method of claim 10 , further comprising ceasing, responsive to the alert, to provide the real-time update of a subset of weights to the plurality of prediction workers.
13. The method of claim 10 , wherein providing the training data to the training worker includes:
generating a hash value using the training data; and
determining the hash value corresponds to the training worker.
14. The method of claim 10 , wherein the observation worker is one of at least two observation workers and the method includes hashing the training data to assign the training data to one of the at least two observation workers.
15. A method comprising:
receiving delta updates from a training worker of a plurality of training workers, each training worker of the plurality of training workers being executable by a processor and being configured to:
store a respective local copy of a machine-learned model,
store a respective snapshot of the machine-learned model,
receive training examples,
train the local copy of the machine-learned model using the training examples, and
periodically send one or more delta updates to a parameter server after training, the delta updates representing differences between the snapshot and the local copy of the machine-learned model or a local state;
aggregating the delta updates into a master copy of the machine-learned model;
providing weights changed in the aggregating to the training worker;
on a first periodic basis:
determining a subset of weights for a real-time update, the subset being less than all of the weights for the master copy and the subset including weights having a change that satisfies a threshold, and
providing the subset of weights as a real-time update to each of a plurality of prediction workers, each of the plurality of prediction workers being executable by a processor and storing a respective local copy of the machine-learned model and running the respective local copy in a serving mode without updating the local copy of the machine-learned model in response to a client request for an output from the model; and
on a second periodic basis, providing a full update of the master copy of the machine-learned model to each of the plurality of prediction workers and to each of the plurality of training workers, wherein the full update is used to replace each respective local copy,
wherein the first periodic basis occurs at least an order of magnitude more frequently than the second periodic basis.
16. The method of claim 15 , wherein the change is a rate of change over time.
17. The method of claim 15 , further comprising:
receiving a request for an initial update from a requesting worker, the requesting worker being one of the plurality of prediction workers;
determining a second set of weights having a change that satisfies a second threshold; and
providing the second set of weights to the requesting worker.
18. The method of claim 17 , wherein the change that satisfies the second threshold is a measure of the absolute value of the weight.
19. The method of claim 17 , wherein the second threshold is set so a fixed percentage of weights are included in the initial update.
20. The method of claim 15 , wherein the real-time update occurs every few seconds and the full update occurs every few hours.
21. The method of claim 15 , wherein providing the subset of weights includes sending the subset to an observation worker, the observation worker running training examples through a respective local copy of the model in a serving mode to generate first metrics and each of the training workers generating second metrics during training and the method further includes:
comparing the first metrics to the second metrics; and
generating an alert when a difference between the first metrics and the second metrics satisfies a second threshold.
22. The method of claim 15 , wherein each of the plurality of prediction workers always run the respective local copy of the model in serving mode and each of the plurality of training workers always run the respective local copy of the model in training mode.
23. The method of claim 15 , wherein the number of prediction workers in the plurality of prediction workers is at least an order of magnitude higher than the number of training workers in the plurality of training workers.