Computer implemented multi-layer federated learning method based on distributed clustering and non-transitory computer readable medium thereof
A multi-layer federated learning method based on distributed clustering is provided, which comprises the following steps. Computing a feature distribution similarity for each of participating nodes with non-(non-independent and identically distributed) data sets, and grouping these nodes into plural clusters by the feature distribution similarity. Updating local model of nodes of each cluster by a federated learning algorithm, and inputting these nodes into a multi-layer aggregation mechanism. Terminating the operation of the multi-layer aggregation mechanism until the clustering result meets a required demand. Furthermore, we implement a blockchain-based multi-layer federated learning system, including model aggregation module, API module, time-series synchronization module, and IPFS, based on distributed clustering architecture. The learning performance is proven to effectively improved.
1 . A computer implemented distributed clustering-based multi-layer federated learning method comprising:
calculating feature distribution similarity of each of a plurality of non-independent and identically distributed (non-IID) data nodes, wherein calculating the feature distribution similarity includes substituting a distribution statistic value of original data;
grouping the non-IID data nodes into an aggregation model including a plurality of clusters based on the feature distribution similarity;
using the feature distribution similarity of each of the non-IID data nodes according to the calculation as a parameter and using a federated learning algorithm, thereby updating a local model of the non-IID data nodes in the aggregation model; and
inputting the aggregation model into an aggregation method of a multi-layer model, wherein the aggregation method of the multi-layer model includes:
a. setting a number of layers of the aggregation model to i, and setting i to 1;
b. aggregating the clusters in an i-th layer and outputting a grouping result;
c. outputting the multi-layered model when the grouping result achieves model performance and stopping execution;
d. when the grouping result does not achieve the model performance, accumulating 1 in i, inputting the grouping result to the i-th layer, and executing step b again and when achieves the model performance, aggregates the clusters into a global cluster;
wherein the feature distribution similarity includes a feature quantity and a feature effect value of each of the non-IID data nodes;
wherein the feature quantity includes a feature quantity with significant differences between data, or a feature quantity with significant correlation within each data;
wherein the feature effect value includes a mean value, a standard deviation or a sample number.
2 . The method of claim 1 , wherein the non-IID data nodes are grouped into a plurality of clusters by using unsupervised learning, in which the unsupervised learning includes K-means, Expectation-Maximization Algorithm, Gaussian Mixture Model or Bayesian Gaussian Mixture Model.
3 . The method of claim 1 , wherein non-IID data nodes of each of the clusters have a similar feature distribution similarity.
4 . The method of claim 1 , wherein the federated learning algorithm is based on each of the clusters by unit, and updates local models of the non-IID data nodes of each of the clusters.
5 . The method of claim 1 , wherein the aggregation method of the multi-layer model is supported and implemented by a blockchain network technology, and the blockchain network technology has a structure including a model aggregation module, an API module, a timing synchronization module, and an interplanetary file system (IPFS).
6 . A non-transitory computer readable medium comprising computer codes stored thereon, the computer codes when executed by one or more processors cause the one or more processors to perform a distributed clustering-based multi-layer federated learning method comprising steps of:
calculating feature distribution similarity of each of a plurality of non-independent and identically distributed (non-IID) data nodes, wherein calculating the feature distribution similarity includes substituting a distribution statistic value of original data;
grouping the non-IID data nodes into an aggregation model including a plurality of clusters based on the feature distribution similarity;
using the feature distribution similarity of each of the non-IID data nodes according to the calculation as a parameter and using a federated learning algorithm, thereby updating a local model of the non-IID data nodes in the aggregation model; and
inputting the aggregation model into an aggregation method of a multi-layer model, wherein the aggregation method of the multi-layer model includes:
a. setting a number of layers of the aggregation model to i, and setting i to 1;
b. aggregating the clusters in an i-th layer and outputting a grouping result;
c. outputting the multi-layered model when the grouping result achieves model performance and stopping execution;
d. when the grouping result does not achieve the model performance, accumulating 1 in i, inputting the grouping result to the i-th layer, and executing step b again and when achieves the model performance, aggregates the clusters into a global cluster;
wherein the feature distribution similarity includes a feature quantity and a feature effect value of each of the non-IID data nodes;
wherein the feature quantity includes a feature quantity with significant differences between data, or a feature quantity with significant correlation within each data;
wherein the feature effect value includes a mean value, a standard deviation or a sample number.
7 . The non-transitory computer readable medium of claim 6 , wherein the non-IID data nodes are grouped into a plurality of clusters by using unsupervised learning, in which the unsupervised learning includes K-means, Expectation-Maximization Algorithm, Gaussian Mixture Model or Bayesian Gaussian Mixture Model.
8 . The non-transitory computer readable medium of claim 6 , wherein non-IID data nodes of each of the clusters have a similar feature distribution similarity.
9 . The non-transitory computer readable medium of claim 6 , wherein the federated learning algorithm is based on each of the clusters by unit, and updates local models of the non-IID data nodes of each of the clusters.
10 . The non-transitory computer readable medium of claim 6 , wherein the aggregation method of the multi-layer model is supported and implemented by a blockchain network technology, and the blockchain network technology has a structure including a model aggregation module, an API module, a timing synchronization module, and an interplanetary file system (IPFS).