Auto tuned swarm learning models
Swarm learning systems and methods can perform hyperparameter tuning. Network nodes train a common machine learning (ML) algorithm using data local to each of the network nodes. Hyperparameter optimization is also performed at each of the network nodes. Model parameters and optimized hyperparameters are shared by the network nodes, and are subsequently merged. An updated version of the common ML algorithm based on the merged model parameters and the merged optimized hyperparameters is generated. That updated version of the common ML algorithm is re-trained at each of the network nodes.
1 . A method comprising:
receiving model parameters, the shared model parameters derived from training local instances of a common machine learning (ML) algorithm at network nodes of a distributed ML network;
receiving optimized hyperparameters associated with the local instances of the common ML algorithm, the optimized hyperparameters resulting from hyperparameter optimization operations performed at each of the network nodes in conjunction with their respective training of the local instances of the common ML algorithm;
merging the model parameters;
merging the optimized hyperparameters;
generating an updated version of the common ML algorithm based on the merged model parameters, and the merged optimized hyperparameters;
generating a common ML model pursuant to re-training instances of the updated version of the common ML algorithm at the network nodes.
2 . The method of claim 1 , wherein the training of the local instances of the common ML algorithm is performed with data local to each of the network nodes.
3 . The method of claim 1 , further comprising, electing a leader node of the network nodes.
4 . The method of claim 3 , wherein the leader node receives the model parameters, and the optimized hyperparameters, and wherein the leader node performs the merging of the model parameters, the merging of the optimized hyperparameters, the generating of the new version of the common ML algorithm, and the generating of the common ML model.
5 . The method of claim 1 , wherein the received model parameters comprise at least one of learnable weights and biases used in the common ML algorithm during the training of the local instances of the common ML algorithm.
6 . The method of claim 1 , wherein the optimized hyperparameters comprise one or more parameters defining at least one of the structure or operation of the common ML algorithm.
7 . The method of claim 1 , wherein the merging of the optimized hyperparameters comprises one of determining a mean value, a median value, or a maximum value representative of the shared optimized hyperparameters.
8 . The method of claim 1 , wherein the receiving of the model parameters and the merging of the model parameters occurs multiple times during the training of the local instances of the common ML algorithm.
9 . The method of claim 8 , wherein the receiving of the optimized hyperparameters and the merging of the optimized hyperparameters occurs once during the training of the local instances of the common ML algorithm.
10 . The method of claim 1 , wherein the re-training of the instances of the new version of the common ML algorithm at the network nodes comprises re-training the instances in accordance with different combinations of the merged optimized hyperparameters.