Fast converging gradient compressor for federated learning
Method for model updating in a federated learning environment, including distributing a current model to client nodes; receiving a first set of gradient sign vectors, wherein each gradient sign vector of the first set of gradient sign vectors is received from one client node; generating a first updated model based on the first set of gradient sign vectors; distributing the first updated model to the plurality of client nodes; storing a first shape parameter and a second shape parameter; receiving, in response to distributing the first updated model, a second set of gradient sign vectors, wherein each gradient sign vector of the second set of gradient sign vectors is received from one client node; generating a second updated model based on the second set of gradient sign vectors, the first shape parameter, and the second shape parameter; and distributing the second updated model to the plurality of client nodes.
1. A method for model updating in a federated learning environment, the method comprising:
distributing, by a model coordinator, a current model with index positions to a plurality of physical client nodes comprising a plurality of neural networks, wherein the current model is trained on the plurality of neural networks using private data in the plurality of physical client nodes producing a gradient vector in each of the plurality of physical client nodes;
receiving, by the model coordinator and in response to distributing the current model, a first set of gradient sign vectors, wherein;
each gradient sign vector of the first set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes,
each gradient sign vector of the first set of gradient sign vectors is based on a sign of the gradient vector for each of the index positions generated by respective physical client nodes of the plurality of physical client nodes using the current model, and
each gradient sign vector initially has a value of zero if the gradient vector is a negative number and a value of one if the gradient vector is zero or a positive number to compress an amount of information transmitted to the model coordinator from each physical client node by compressing each index position to a single bit, which reduces a network bandwidth necessary to transmit the first set of gradient sign vectors;
generating, by the model coordinator, a first updated model by using a Bayesian Beta-Bernoulli model to determine an expected value for each of the index positions with inputs comprising the first set of gradient sign vectors, a first shape parameter, and a second shape parameter;
distributing the first updated model to the plurality of physical client nodes;
storing, by the model coordinator, the first shape parameter and the second shape parameter;
receiving, by the model coordinator and in response to distributing the first updated model, a second set of gradient sign vectors, wherein each gradient sign vector of the second set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes;
generating, by the model coordinator, a second updated model based on the second set of gradient sign vectors, the first shape parameter, and the second shape parameter; and
distributing the second updated model to the plurality of physical client nodes.
2. The method of claim 1 , further comprising:
updating the first shape parameter using a set of gradient sign values at a gradient sign vector index position within the second set of gradient sign vectors to obtain an updated first shape parameter in a Bayesian update;
updating the second shape parameter using the set of gradient sign values at the gradient sign vector index position within the second set of gradient sign vectors to obtain an updated second shape parameter in a Bayesian update;
receiving, by the model coordinator and in response to distributing the second updated model, a third set of gradient sign vectors, wherein each gradient sign vector of the third set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes; and
generating, by the model coordinator, a third updated model using the second set of gradient sign vectors, the updated first shape parameter, and the updated second shape parameter.
3. The method of claim 2 , wherein:
updating the first shape parameter using the set of gradient sign values comprises adding a quantity of positive gradient sign values of the set of gradient sign values to the first shape parameter, and
updating the second shape parameter using the set of gradient sign values comprises adding a quantity of negative gradient sign values of the set of gradient sign values to the second shape parameter.
4. The method of claim 2 , further comprising:
making a determination, by the model coordinator and after distributing the second updated model, that a cycle threshold is reached;
discarding, based on the determination, the updated first shape parameter and the updated second shape parameter; and
using the first shape parameter, the second shape parameter, and a next set of gradient vectors from the plurality of physical client nodes to generate the next updated model.
5. The method of claim 1 , wherein, when the current model is an initial model, generating the first updated model comprises:
calculating an average gradient index position value for each gradient index position of the first updated model using the first set of gradient sign vectors.
6. The method of claim 1 , further comprising, before generating the first updated model:
decoding each gradient sign vector of the first set of gradient sign vectors to include only non-zero real numbers to obtain a set of decoded gradient sign vectors,
wherein the model coordinator generates the first updated model using the set of decoded gradient sign vectors.
7. The method of claim 1 , wherein the first shape parameter and the second shape parameter are equal to one.
8. The method of claim 1 , wherein prior to generating a first updated model, each gradient sign vector of the first set of gradient sign vectors is decoded where a value of zero is changed to negative one and a value of one is changed to positive one.
9. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for model updating in a federated learning environment, the method comprising:
distributing, by a model coordinator, a current model with index positions to a plurality of physical client nodes comprising a plurality of neural networks, wherein the current model is trained on the plurality of neural networks using private data in the plurality of physical client nodes producing a gradient vector in each of the plurality of physical client nodes;
receiving, by the model coordinator and in response to distributing the current model, a first set of gradient sign vectors, wherein:
each gradient sign vector of the first set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes,
each gradient sign vector of the first set of gradient sign vectors is based on a sign of the gradient vector for each of the index positions generated by respective physical client nodes of the plurality of physical client nodes using the current model, and
each gradient sign vector initially has a value of zero if the gradient vector is a negative number and a value of one if the gradient vector is zero or a positive number to compress an amount of information transmitted to the model coordinator from each physical client node by compressing each index position to a single bit, which reduces a network bandwidth necessary to transmit the first set of gradient sign vectors;
generating, by the model coordinator, a first updated model by using a Bayesian Beta-Bernoulli model to determine an expected value for each of the index positions with inputs comprising the first set of gradient sign vectors, a first shape parameter, and a second shape parameter;
distributing the first updated model to the plurality of physical client nodes;
storing, by the model coordinator, the first shape parameter and the second shape parameter;
receiving, by the model coordinator and in response to distributing the first updated model, a second set of gradient sign vectors, wherein each gradient sign vector of the second set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes;
generating, by the model coordinator, a second updated model based on the second set of gradient sign vectors, the first shape parameter, and the second shape parameter; and
distributing the second updated model to the plurality of physical client nodes.
10. The non-transitory computer readable medium of claim 9 , wherein the method performed by executing the computer readable program code further comprises:
updating the first shape parameter using a set of gradient sign values at a gradient sign vector index position within the second set of gradient sign vectors to obtain an updated first shape parameter in a Bayesian update;
updating the second shape parameter using the set of gradient sign values at the gradient sign vector index position within the second set of gradient sign vectors to obtain an updated second shape parameter in a Bayesian update;
receiving, by the model coordinator and in response to distributing the second updated model, a third set of gradient sign vectors, wherein each gradient sign vector of the third set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes; and
generating, by the model coordinator, a third updated model using the second set of gradient sign vectors, the updated first shape parameter, and the updated second shape parameter.
11. The non-transitory computer readable medium of claim 10 , wherein:
updating the first shape parameter using the set of gradient sign values comprises adding a quantity of positive gradient sign values of the set of gradient sign values to the first shape parameter, and
updating the second shape parameter using the set of gradient sign values comprises adding a quantity of negative gradient sign values of the set of gradient sign values to the second shape parameter.
12. The non-transitory computer readable medium of claim 10 , wherein the method performed by executing the computer readable program code further comprises:
making a determination, by the model coordinator and after distributing the second updated model, that a cycle threshold is reached;
discarding, based on the determination, the updated first shape parameter and the updated second shape parameter; and
using the first shape parameter, the second shape parameter, and a next set of gradient vectors from the plurality of physical client nodes to generate the next updated model.
13. The non-transitory computer readable medium of claim 9 , wherein, when the current model is an initial model, generating the first updated model comprises:
calculating an average gradient index position value for each gradient index position of the first updated model using the first set of gradient sign vectors.
14. The non-transitory computer readable medium of claim 9 , wherein the method performed by executing the computer readable program code further comprises, before generating the first updated model:
decoding each gradient sign vector of the first set of gradient sign vectors to include only non-zero real numbers to obtain a set of decoded gradient sign vectors,
wherein the model coordinator generates the first updated model using the set of decoded gradient sign vectors.
15. The non-transitory computer readable medium of claim 9 , wherein the first shape parameter and the second shape parameter are equal to one.
16. The non-transitory computer readable medium of claim 9 , wherein prior to generating a first updated model, each gradient sign vector of the first set of gradient sign vectors is decoded where a value of zero is changed to negative one and a value of one is changed to positive one.
17. A system for model updating in a federated learning environment, the system comprising:
a model coordinator, executing on a processor comprising circuitry, and configured to:
distribute a current model with index positions to a plurality of physical client nodes comprising a plurality of neural networks, wherein the current model is trained on the plurality of neural networks using private data in the plurality of physical client nodes producing a gradient vector in each of the plurality of physical client nodes;
receive, in response to distributing the current model, a first set of gradient sign vectors, wherein:
each gradient sign vector of the first set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes,
each gradient sign vector of the first set of gradient sign vectors is based on a sign of the gradient vector for each of the index positions generated by respective physical client nodes of the plurality of physical client nodes using the current model, and
each gradient sign vector initially has a value of zero if the gradient vector is a negative number and a value of one if the gradient vector is zero or a positive number to compress an amount of information transmitted to the model coordinator from each physical client node by compressing each index position to a single bit, which reduces a network bandwidth necessary to transmit the first set of gradient sign vectors;
generate a first updated model by using a Bayesian Beta-Bernoulli model to determine an expected value for each of the index positions with inputs comprising the first set of gradient sign vectors, a first shape parameter, and a second shape parameter;
distribute the first updated model to the plurality of physical client nodes;
store the first shape parameter and the second shape parameter;
receive, in response to distributing the first updated model, a second set of gradient sign vectors, wherein each gradient sign vector of the second set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes;
generate a second updated model based on the second set of gradient sign vectors, the first shape parameter, and the second shape parameter; and
distribute the second updated model to the plurality of physical client nodes.
18. The system of claim 17 , wherein the model coordinator is further configured to:
update the first shape parameter using a set of gradient sign values at a gradient sign vector index position within the second set of gradient sign vectors to obtain an updated first shape parameter in a Bayesian update;
update the second shape parameter using the set of gradient sign values at the gradient sign vector index position within the second set of gradient sign vectors to obtain an updated second shape parameter in a Bayesian update;
receive, by the model coordinator and in response to distributing the second updated model, a third set of gradient sign vectors, wherein each gradient sign vector of the third set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes; and
generate, by the model coordinator, a third updated model using the second set of gradient sign vectors, the updated first shape parameter, and the updated second shape parameter.
19. The system of claim 18 , wherein the model coordinator is further configured to:
make a determination, after distributing the second updated model, that a cycle threshold is reached;
discard, based on the determination, the updated first shape parameter and the updated second shape parameter; and
use the first shape parameter, the second shape parameter, and a next set of gradient vectors from the plurality of physical client nodes to generate the next updated model.
20. The system of claim 17 , wherein the first shape parameter and the second shape parameter are equal to one.