Stochastic layer-wise averaging aggregation for federated learning
One method includes stochastically selecting, by a central node, a subset of edge nodes from a group of edge nodes that collectively defines a federation, querying, by the central node, the edge nodes of the subset for updates to a global model maintained by the central node, receiving, by the central node from the edge nodes of the subset, respective updates to one or more layers of the global model, and updating, by the central node, the global model, using the updates received from the edge nodes of the subset.
1 . A method, comprising:
stochastically selecting, by a central node, using a processor-implemented random sampling algorithm that populates a binary participation matrix defining node participation on a per-layer basis, a subset of edge nodes from a group of edge nodes that collectively defines a federation, wherein the participation matrix is constrained such that (i) each layer of a multi-layer global model has at least one participating edge node, and (ii) each edge node participates in updates for no fewer than a minimum number of layers p_min and no more than a maximum number of layers p_max;
querying, by the central node, the edge nodes of the subset for only those layers for which the participation matrix indicates participation for updates to a global model maintained by the central node;
receiving, by the central node from the edge nodes of the subset, respective updates comprising complete layer-wise parameter gradients for the indicated layers and excluding gradients for non-indicated layers, to one or more layers of the global model; and
updating, by the central node, the global model, on a layer-by-layer basis, using the updates received from the edge nodes of the subset, by aggregating only the received layer-wise gradients so as to produce updated layers while withholding sufficient gradient information to resist model inversion attacks while maintaining convergence of the global model.
2 . The method as recited in claim 1 , wherein there is a specified minimum and/or maximum number of nodes in the subset.
3 . The method as recited in claim 1 , wherein the global model is a machine learning model.
4 . The method as recited in claim 1 , wherein the updates comprise layers of one or more local instances of the global model running at the edge nodes.
5 . The method as recited in claim 1 , wherein each node that contributes an update transmits fewer than all layers of a model instance running at that node.
6 . The method as recited in claim 1 , wherein the updates each comprise one or more gradients.
7 . The method as recited in claim 1 , wherein the selecting, querying, receiving, and updating, are each performed for ‘n’ federated learning rounds until the global model converges, and ‘n’ is any integer equal to, or greater than, 1.
8 . The method as recited in claim 1 , wherein the updating comprises averaging respective parameter values of the updates to define respective parameter values for one or more layers of the global model.
9 . The method as recited in claim 1 , wherein the edge nodes selected for inclusion in the subset change from one round of federated learning training to another round of the federated learning training.
10 . The method as recited in claim 1 , wherein each node in the subset is limited to a number of times that that node will be queried for updates.
11 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
stochastically selecting, by a central node, using a processor-implemented random sampling algorithm that populates a binary participation matrix defining node participation on a per-layer basis, a subset of edge nodes from a group of edge nodes that collectively defines a federation, wherein the participation matrix is constrained such that (i) each layer of a multi-layer global model has at least one participating edge node, and (ii) each edge node participates in updates for no fewer than a minimum number of layers p_min and no more than a maximum number of layers p_max;
querying, by the central node, the edge nodes of the subset for only those layers for which the participation matrix indicates participation for updates to a global model maintained by the central node;
receiving, by the central node from the edge nodes of the subset, respective updates comprising complete layer-wise parameter gradients for the indicated layers and excluding gradients for non-indicated layers, to one or more layers of the global model; and
updating, by the central node, the global model, on a layer-by-layer basis, using the updates received from the edge nodes of the subset, by aggregating only the received layer-wise gradients so as to produce updated layers while withholding sufficient gradient information to resist model inversion attacks while maintaining convergence of the global model.
12 . The non-transitory storage medium as recited in claim 11 , wherein there is a specified minimum and/or maximum number of nodes in the subset.
13 . The non-transitory storage medium as recited in claim 11 , wherein the global model is a machine learning model.
14 . The non-transitory storage medium as recited in claim 11 , wherein the updates comprise layers of one or more local instances of the global model running at the edge nodes.
15 . The non-transitory storage medium as recited in claim 11 , wherein each node that contributes an update transmits fewer than all layers of a model instance running at that node.
16 . The non-transitory storage medium as recited in claim 11 , wherein the updates each comprise one or more gradients.
17 . The non-transitory storage medium as recited in claim 11 , wherein the selecting, querying, receiving, and updating, are each performed for ‘n’ federated learning rounds until the global model converges, and ‘n’ is any integer equal to, or greater than, 1.
18 . The non-transitory storage medium as recited in claim 11 , wherein the updating comprises averaging respective parameter values of the updates to define respective parameter values for one or more layers of the global model.
19 . The non-transitory storage medium as recited in claim 11 , wherein the edge nodes selected for inclusion in the subset change from one round of federated learning training to another round of the federated learning training.
20 . The non-transitory storage medium as recited in claim 11 , wherein each node in the subset is limited to a number of times that that node will be queried for updates.