Communication efficient federated learning
The present disclosure provides efficient communication techniques for transmission of model updates within a machine learning framework, such as, for example, a federated learning framework in which a high-quality centralized model is trained on training data distributed overt a large number of clients each with unreliable network connections and low computational power. In an example federated learning setting, in each of a plurality of rounds, each client independently updates the model based on its local data and communicates the updated model back to the server, where all the client-side updates are used to update a global model. The present disclosure provides systems and methods that reduce communication costs. In particular, the present disclosure provides at least: structured update approaches in which the model update is restricted to be small and sketched update approaches in which the model update is compressed before sending to the server.
1. A client computing device, comprising:
at least one processor; and
at least one non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the client computing device to perform operations, the operations comprising:
obtaining global values for a set of parameters of a machine-learned model;
training the machine-learned model based at least in part on a local dataset to obtain an update matrix that is descriptive of updated values for the set of parameters of the machine-learned model, wherein the local dataset is stored locally by the client computing device;
encoding the update matrix to obtain an encoded update, wherein encoding the update matrix comprises multiplying a vector of the update matrix by a rotation matrix to obtain a rotated update, and wherein the rotation matrix is a structured rotation matrix that does not require complete generation of the rotation matrix by the client computing device; and
communicating the encoded update to a server computing device.
2. The client computing device of claim 1 , wherein encoding the update matrix comprises subsampling the update matrix to obtain the encoded update.
3. The client computing device of claim 2 , wherein subsampling the update matrix comprises:
generating a parameter mask that specifies a portion of the set of parameters to be sampled; and
subsampling the update matrix according to the parameter mask.
4. The client computing device of claim 3 , wherein generating the parameter mask comprises generating the parameter mask based at least in part on a seed and a pseudorandom number generator, wherein both the client computing device and the server computing device have knowledge of the seed such that the parameter mask is reproducible by the server computing device.
5. The client computing device of claim 1 , wherein encoding the update matrix comprises probabilistically quantizing one or more values included in the update matrix.
6. The client computing device of claim 1 , wherein encoding the update matrix comprises performing probabilistic binary quantization for one or more values included in the update matrix to change each of the one or more values to a maximum value included in the update matrix or a minimum value included in the update matrix.
7. The client computing device of claim 1 , wherein encoding the update matrix comprises:
defining a plurality of intervals between a maximum value included in the update matrix and a minimum value included in the update matrix; and
probabilistically changing each of one or more values included in the update matrix to a local interval maximum or a local interval maximum.
8. The client computing device of claim 1 , wherein encoding the update matrix further comprises: probabilistically quantizing one or more values included in the rotated update.
9. At least one non-transitory computer-readable medium that stores instructions that, when executed by a client computing device, cause the client computing device to perform operations, the operations comprising:
obtaining global values for a set of parameters of a machine-learned model;
training the machine-learned model based at least in part on a local dataset to obtain an update matrix that is descriptive of updated values for the set of parameters of the machine-learned model, wherein the local dataset is stored locally by the client computing device, and wherein the update matrix is restricted to be at least one of a low-rank matrix and a sparse matrix;
encoding the update matrix to obtain an encoded update, wherein encoding the update matrix comprises multiplying a vector of the update matrix by a rotation matrix to obtain a rotated update, and wherein the rotation matrix is a structured rotation matrix that does not require complete generation of the rotation matrix by the client computing device; and
communicating the encoded update to a server computing device.
10. A computer-implemented method, comprising:
obtaining, by a client computing device comprising one or more processors, global values for a set of parameters of a machine-learned model;
training, by the client computing device, the machine-learned model based at least in part on a local dataset to obtain an update matrix that is descriptive of updated values for the set of parameters of the machine-learned model, wherein the local dataset is stored locally by the client computing device;
encoding, by the client computing device, the update matrix to obtain an encoded update, wherein encoding the update matrix comprises multiplying a vector of the update matrix by a rotation matrix to obtain a rotated update, and wherein the rotation matrix is a structured rotation matrix that does not require complete generation of the rotation matrix by the client computing device; and
communicating, by the client computing device, the encoded update to a server computing device.
11. The method of claim 10 , wherein encoding the update matrix comprises subsampling the update matrix to obtain the encoded update.
12. The method of claim 11 , wherein subsampling the update matrix comprises: generating, by the client computing device, a parameter mask that specifies a portion of the set of parameters to be sampled; and subsampling, by the client computing device, the update matrix according to the parameter mask.
13. The method of claim 12 , wherein generating the parameter mask comprises generating, by the client computing device, the parameter mask based at least in part on a seed and a pseudo-random number generator, wherein both the client computing device and the server computing device have knowledge of the seed such that the parameter mask is reproducible by the server computing device.
14. The method of claim 10 , wherein encoding the update matrix comprises probabilistically quantizing, by the client computing device, one or more values included in the update matrix.
15. The method of claim 10 , wherein encoding the update matrix comprises performing, by the client computing device, probabilistic binary quantization for one or more values included in the update matrix to change each of the one or more values to a maximum value included in the update matrix or a minimum value included in the update matrix.
16. The method of claim 10 , wherein encoding the update matrix comprises:
defining, by the client computing device, a plurality of intervals between a maximum value included in the update matrix and a minimum value included in the update matrix; and
probabilistically changing, by the client computing device, each of one or more values included in the update matrix to a local interval maximum or a local interval maximum.
17. The method of claim 10 , wherein encoding the update matrix further comprises: probabilistically quantizing, by the client computing device, one or more values included in the rotated update.