K-QUANT GRADIENT COMPRESSOR FOR FEDERATED LEARNING
Techniques described herein relate to a method for model updating in a federated learning environment. The method may include distributing, by a model coordinator, a current model to a plurality of client nodes; receiving, by the model coordinator and in response to distributing the current model, a set of gradient K-quant vectors, wherein each gradient K-quant vector of the first set of gradient K-quant vectors is received from one client node of the plurality of client nodes. The gradient K-quant vectors may be compressed representations of gradient vectors. The compression may be performed by determining a bin index value corresponding to the gradient vector values, based on a K value and range received from the model coordinator. The model coordinator may use the gradient K-quant vectors to generate an updated model, and send the updated model to the client nodes for use in the next training cycle.
1 . A method for model updating in a federated learning environment, the method comprising:
distributing, by a model coordinator, a current model to a plurality of client nodes;
receiving, by the model coordinator and in response to distributing the current model, a first set of gradient K-quant vectors, wherein each gradient K-quant vector of the first set of gradient K-quant vectors is received from one client node of the plurality of client nodes;
generating, by the model coordinator, a first updated model based on the first set of gradient K-quant vectors;
distributing the first updated model to the plurality of client nodes;
storing, by the model coordinator, a plurality of shape parameters;
receiving, by the model coordinator and in response to distributing the first updated model, a second set of gradient K-quant vectors, wherein each gradient K-quant vector of the second set of gradient K-quant vectors is received from one client node of the plurality of client nodes;
generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters; and
distributing the second updated model to the plurality of client nodes.
2 . The method of claim 1 , further comprising:
updating a first shape parameter of the plurality of shape parameters using a set of gradient K-quant values at a gradient K-quant vector position within the second set of gradient K-quant vectors to obtain an updated first shape parameter;
receiving, by the model coordinator and in response to distributing the second updated model, a third set of gradient K-quant vectors, wherein each gradient K-quant vector of the third set of gradient K-quant vectors is received from one client node of the plurality of client nodes; and
generating, by the model coordinator, a third updated model using at least the second set of gradient K-quant vectors and the updated first shape parameter.
3 . The method of claim 2 , further comprising:
making a determination, by the model coordinator and after distributing the second updated model, that a cycle threshold is reached;
discarding, based on the determination; and
using the plurality of shape parameters and a next set of gradient K-quant vectors from the plurality of client nodes to generate a next updated model.
4 . The method of claim 1 , wherein, before receiving the first set of gradient K-quant vectors, the method further comprises distributing, by the model coordinator and to the plurality of client nodes, a K value and a range, wherein the plurality of client nodes use the K value and the range to generate the first set of gradient K-quant vectors.
5 . The method of claim 1 , wherein, when the current model is an initial model, generating the first updated model comprises:
calculating an average gradient position value for each gradient position of the first updated model using the first set of gradient K-quant vectors.
6 . The method of claim 1 , further comprising, before generating the first updated model:
decoding each gradient K-quant vector of the first set of gradient K-quant vectors to obtain a plurality of bin index values.
wherein the model coordinator generates the first updated model using the plurality of bin index values.
7 . The method of claim 6 , wherein the plurality of bin index values are used to obtain a plurality of mean range values of a range associated with each of the plurality of bin index values.
8 . A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for model updating in a federated learning environment, the method comprising:
distributing, by a model coordinator, a current model to a plurality of client nodes;
receiving, by the model coordinator and in response to distributing the current model, a first set of gradient K-quant vectors, wherein each gradient K-quant vector of the first set of gradient K-quant vectors is received from one client node of the plurality of client nodes;
generating, by the model coordinator, a first updated model based on the first set of gradient K-quant vectors;
distributing the first updated model to the plurality of client nodes;
storing, by the model coordinator, a plurality of shape parameters;
receiving, by the model coordinator and in response to distributing the first updated model, a second set of gradient K-quant vectors, wherein each gradient K-quant vector of the second set of gradient K-quant vectors is received from one client node of the plurality of client nodes;
generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters; and
distributing the second updated model to the plurality of client nodes.
9 . The non-transitory computer readable medium of claim 8 , wherein the method performed by executing the computer readable program code further comprises:
updating a first shape parameter of the plurality of shape parameters using a set of gradient K-quant values at a gradient K-quant vector position within the second set of gradient K-quant vectors to obtain an updated first shape parameter;
receiving, by the model coordinator and in response to distributing the second updated model, a third set of gradient K-quant vectors, wherein each gradient K-quant vector of the third set of gradient K-quant vectors is received from one client node of the plurality of client nodes; and
generating, by the model coordinator, a third updated model using at least the second set of gradient K-quant vectors and the updated first shape parameter.
10 . The non-transitory computer readable medium of claim 9 , wherein the method performed by executing the computer readable program code further comprises:
making a determination, by the model coordinator and after distributing the second updated model, that a cycle threshold is reached;
discarding, based on the determination; and
using the plurality of shape parameters and a next set of gradient K-quant vectors from the plurality of client nodes to generate a next updated model.
11 . The non-transitory computer readable medium of claim 8 , wherein, before receiving the first set of gradient K-quant vectors, the method further comprises distributing, by the model coordinator and to the plurality of client nodes, a K value and a range, wherein the plurality of client nodes use the K value and the range to generate the first set of gradient K-quant vectors.
12 . The non-transitory computer readable medium of claim 8 , wherein, when the current model is an initial model, generating the first updated model comprises:
calculating an average gradient position value for each gradient position of the first updated model using the first set of gradient K-quant vectors.
13 . The non-transitory computer readable medium of claim 8 , wherein the method performed by executing the computer readable program code further comprises, before generating the first updated model:
decoding each gradient K-quant vector of the first set of gradient K-quant vectors to obtain a plurality of bin index values.
wherein the model coordinator generates the first updated model using the plurality of bin index values.
14 . The non-transitory computer readable medium of claim 13 , wherein the plurality of bin index values are used to obtain a plurality of mean range values of a range associated with each of the plurality of bin index values.
15 . A system for model updating in a federated learning environment, the system comprising:
a model coordinator, executing on a processor comprising circuitry, and configured to:
distribute a current model to a plurality of client nodes;
receive, in response to distributing the current model, a first set of gradient K-quant vectors, wherein each gradient K-quant vector of the first set of gradient K-quant vectors is received from one client node of the plurality of client nodes;
generate a first updated model based on the first set of gradient K-quant vectors;
distribute the first updated model to the plurality of client nodes;
store a plurality of shape parameters;
receive, in response to distributing the first updated model, a second set of gradient K-quant vectors, wherein each gradient K-quant vector of the second set of gradient K-quant vectors is received from one client node of the plurality of client nodes;
generate a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters; and
distribute the second updated model to the plurality of client nodes.
16 . The system of claim 15 , wherein the model coordinator is further configured to:
update a first shape parameter of the plurality of shape parameters using a set of gradient K-quant values at a gradient K-quant vector position within the second set of gradient K-quant vectors to obtain an updated first shape parameter;
receive, in response to distributing the second updated model, a third set of gradient K-quant vectors, wherein each gradient K-quant vector of the third set of gradient K-quant vectors is received from one client node of the plurality of client nodes; and
generate a third updated model using at least the second set of gradient K-quant vectors and the updated first shape parameter.
17 . The system of claim 16 , wherein the model coordinator is further configured to:
making a determination, by the model coordinator and after distributing the second updated model, that a cycle threshold is reached;
discarding, based on the determination; and
using the plurality of shape parameters and a next set of gradient K-quant vectors from the plurality of client nodes to generate a next updated model.
18 . The system of claim 15 , wherein, before receiving the first set of gradient K-quant vectors, the model coordinator is further configured to distribute, to the plurality of client nodes, a K value and a range, wherein the plurality of client nodes use the K value and the range to generate the first set of gradient K-quant vectors.
19 . The system of claim 15 , further comprising, before generating the first updated model:
decoding each gradient K-quant vector of the first set of gradient K-quant vectors to obtain a plurality of bin index values.
wherein the model coordinator generates the first updated model using the plurality of bin index values.
20 . The system of claim 19 , wherein the plurality of bin index values are used to obtain a plurality of mean range values of a range associated with each of the plurality of bin index values.