Adaptively synchronizing learning of multiple learning models
A system and a method for adaptively synchronizing learning of multiple learning models are disclosed. Several local learning models are executed on multiple nodes. Learning model parameters are shared by such nodes to a master node, in multiple iterations, after a predefined synchronization interval. Such learning model parameters are aggregated and central learning models are generated based on aggregated set of learning model parameters. Accuracies of the central learning models and an average accuracy of the central learning models are determined. Accuracy of an immediate central learning model i.e. the one received after determining the average accuracy, is compared with the average accuracy. Based on the difference between the accuracy of the immediate central learning model and the average accuracy, the synchronization interval is modified, and the multiple nodes are updated about this modified synchronization interval.
1. A system, comprising:
a memory;
a processor that is configured to execute machine readable instructions stored in the memory to transmit communications between a plurality of data processing nodes and a master node;
the plurality of data processing nodes programmed to run a plurality of local learning models; and
the master node programmed to communicate with the plurality of data processing nodes to:
iteratively receive, by the master node over a predefined synchronization interval, a set of learning model weights from the plurality of data processing nodes, wherein the predefined synchronization interval is stored as an integer value at the master node;
aggregate, by the master node, the set of learning model weights received iteratively from the plurality of data processing nodes, after a predefined synchronization interval, wherein the set of learning model weights are received for a predefined number of iterations;
generate central learning models based on the aggregated set of learning model weights received after each of the predefined number of iterations;
generate an average accuracy value associated with the central learning models;
determine a difference between an accuracy of a central learning model and the average accuracy value;
update an accuracy metric based on the difference between the accuracy of the central model and the average accuracy value;
compare the accuracy metric to a range established by a set of thresholds; and
when the accuracy metric is outside of the range established by the set of thresholds, downscale, in accordance with the integer value, the predefined synchronization interval by a predefined scaling factor that increases sharing of the set of learning model weights by the data processing nodes.
2. The system of claim 1 , wherein the accuracy metric is represented as Mean Squared Errors (MSEs).
3. The system of claim 1 , wherein the average accuracy value associated with the central learning models and the accuracy of the central learning model are determined based on results obtained by implementation of the central learning models and the central learning model on pre-stored evaluation data.
4. The system of claim 1 , wherein the predefined synchronization interval is up-scaled by the predefined scaling factor when the accuracy metric exceeds the range established by the set of thresholds.
5. The system of claim 1 , wherein the master node and the plurality of data processing nodes are a part of a decentralized data processing network or a distributed data processing network.
6. A method, comprising:
periodically receiving, at a master node of a plurality of data processing nodes, a set of learning model weights from each node of the plurality of data processing nodes,
wherein the each successive set of learning model weights from the respective nodes are iteratively received after a predefined synchronization interval, wherein the predefined synchronization interval is stored as an integer value at the master node, and wherein the set of learning model weights are received for a predefined number of iterations;
aggregating, after each of the predefined number of iterations, the set of learning model weights received after the predefined synchronization interval;
generating central learning models based on the aggregated set of learning model weights received after each of the predefined number of iterations;
generating an average accuracy value associated with the central learning models;
determining a difference between an accuracy of a central learning model and the average accuracy value;
comparing the difference in accuracy to a set of thresholds;
when the difference in accuracy is outside of a range established by the set of thresholds, downscaling, in accordance with the integer value, the predefined synchronization interval by a predefined scaling factor that increases sharing of the set of learning model weights by the data processing nodes modifying the predefined synchronization interval; and
transmitting, to the plurality of data processing nodes, the modified synchronization interval.
7. The method of claim 6 , wherein the accuracy is represented as Mean Squared Errors (MSEs).
8. The method of claim 6 , wherein the average accuracy value associated with the central learning models and the accuracy of the central learning model are determined based on results obtained by implementation of the central learning models and the central learning model on pre-stored evaluation data.
9. The method of claim 6 , wherein the predefined synchronization interval is modified by the master node elected amongst the plurality of data processing nodes.
10. The method of claim 6 , wherein the synchronization interval is up-scaled by the predefined scaling factor when the difference in accuracy exceeds the range established by the set of thresholds.
11. The method of claim 6 , wherein the master node and the plurality of data processing nodes are a part of a decentralized data processing network or a distributed data processing network.
12. A non-transitory machine-readable storage medium that stores instructions which, when executed by a processor, perform a plurality of operations comprising:
receiving a set of learning model weights iteratively from each node of a plurality of data processing nodes, wherein the set of learning model weights is received after a predefined synchronization interval, wherein the predefined synchronization interval is stored as an integer value at the master node, and wherein the set of learning model weights are received for a predefined number of iterations;
generating central learning models based on the set of learning model weights received after each of the predefined number of iterations;
generating an average accuracy value associated with the central learning models;
determining a difference between an accuracy of a central learning model and the average accuracy value;
comparing the difference in accuracy to a range established by a set of thresholds;
when the difference in accuracy is outside of the range, downscaling, in accordance with the integer value, the predefined synchronization interval by a predefined scaling factor that increases sharing of the set of learning model weights by the data processing nodes; and
transmitting, to the plurality of data processing nodes, the modified predefined synchronization interval.
13. The system of claim 1 , wherein the set of learning model weights indicate a level of learning achieved by the plurality of data processing nodes.
14. The system of claim 1 , wherein the master node is further programmed to:
receive pre-stored evaluation data;
generate, using the central learning model, an inferred label of the pre-stored evaluation data; and
store the inferred label of the pre-stored evaluation data with the master node.
15. The system of claim 1 , wherein the master node is further programmed to:
identify common data in the set of learning model weights.
16. The system of claim 1 , wherein the master node is further programmed to:
adjust the predefined synchronization interval by the predefined scaling factor to reduce a frequency of synchronization of the plurality of local learning models with the master node.
17. The system of claim 1 , wherein the master node is further programmed to:
plot a learning curve that illustrates results obtained upon synchronization of the set of learning models.
18. The system of claim 1 , wherein the plurality of data processing nodes use an Application Programming Interface (API) to communicate with the master node.
19. The system of claim 1 , wherein the set of learning model weights is averaged and aggregated by the master node.
20. The system of claim 1 , wherein the master node is further programmed to:
determine that the predefined synchronization interval has expired for the iteration; and
in response to the predefined synchronization interval expiring, trigger the aggregation and averaging of the set of learning model weights.