Fly parameter compression and decompression to facilitate forward and/or back propagation at clients during federated learning
Implementations are directed to efficient federated learning of machine learning (ML) model(s) through on-the-fly decompression and compression of model parameters, of the ML model(s), when facilitating forward propagation and/or back propagation at client device(s). For example, implementations can transmit, from a remote system to a client device, a compressed on-device ML model that includes some compressed parameters. Further, the client device can, in performing forward propagation and/or back propagation using the on-device ML model, decompress those compressed parameters on-the-fly as the parameters are needed for the propagation. The propagation will utilize the decompressed parameters that were decompressed on the fly. Further, after the decompressed parameters are utilized, they can be deallocated from memory (while their compressed counterparts optionally remain in memory) to enable allocation of memory for further decompressed parameters that will be needed next and/or needed for other ongoing process(es).
1 . A method implemented by one or more processors of a client device, the method comprising:
receiving, from a remote system, a client machine learning (ML) model in a compressed format,
wherein the remote system hosts a global ML model that corresponds to the client ML model, and the client ML model includes compressed model parameters for processing model input data;
for each of multiple compressed subsets of the compressed model parameters of the client ML model, to facilitate forward propagation, and while all other of the compressed subsets of parameters of the client ML model remain compressed:
decompressing the compressed subset of parameters to generate a corresponding decompressed subset of parameters; and
processing corresponding data using the corresponding decompressed subset of parameters to generate a next iteration of the corresponding data and in furtherance of determining a predicted output from the forward propagation;
determining, based on the predicted output from the forward propagation, a loss in furtherance of updating the compressed model parameters;
for each of multiple compressed subsets of the compressed model parameters of the client ML model, to facilitate back propagation, and while all other of the compressed subsets of parameters of the client ML model remain compressed:
decompressing the compressed subset of parameters to generate the corresponding decompressed subset of parameters;
processing a remaining portion of the loss and the corresponding decompressed subset of parameters to generate corresponding decompressed updated parameters for the corresponding decompressed subset of parameters;
compressing the corresponding decompressed updated parameters to generate corresponding updated compressed parameters for the client ML model; and
storing the corresponding updated compressed parameters; and
providing, to the remote system from the client device, the corresponding compressed updated parameters for the client ML model,
wherein the global ML model is updated by at least decompressing, by the remote system, the corresponding compressed updated parameters from the client device.
2 . The method of claim 1 , wherein decompressing the compressed subset of parameters, for a given one of the compressed subsets, to facilitate forward propagation, includes deallocating client device memory for an additional one of the decompressed subsets utilized in an immediately preceding iteration of the forward propagation.
3 . The method of claim 2 , wherein decompressing the compressed subset of parameters, for the given one of the compressed subsets, to facilitate back propagation, includes deallocating client device memory for a further one of the decompressed subsets utilized in an immediately preceding iteration of the back propagation.
4 . The method of claim 1 , wherein the client ML model includes non-compressed model parameters that are also utilized to facilitate forward propagation and/or back propagation.
5 . The method of claim 1 , wherein the client ML model is utilized by the client device to facilitate automatic speech recognition (ASR) and an initial instance of the corresponding data is based on audio captured by the client device or another computing device.
6 . The method of claim 1 , wherein decompressing the compressed subset of parameters comprises:
applying a corresponding scaling factor and/or a corresponding biasing value to each parameter of the decompressed subset of parameters.
7 . The method of claim 1 wherein determining the loss is based on comparing the predicted output to a ground truth value.
8 . A method implemented by one or more processors of a client device, the method comprising:
receiving, from a remote system, a client machine learning (ML) model in a compressed format,
wherein the remote system hosts a global ML model that corresponds to the client ML model, and the client ML model includes compressed model parameters for processing model input data;
for each of multiple compressed subsets of the compressed model parameters of the client ML model, to facilitate forward propagation, and while all other of the compressed subsets of parameters of the client ML model remain compressed:
decompressing the compressed subset of parameters to generate a corresponding decompressed subset of parameters; and
processing corresponding data using the corresponding decompressed subset of parameters to generate a next iteration of the corresponding data and in furtherance of determining a predicted output from the forward propagation;
determining, based on the predicted output from the forward propagation, a loss in furtherance of updating the compressed model parameters; and
providing, to the remote system, compressed updated parameters that are for the client ML model and that are generated based on the loss,
wherein providing the compressed updated parameters to the remote system causes the remote system to update the global ML model based on a decompression of the compressed updated parameters.
9 . The method of claim 8 , wherein decompressing the compressed subset of parameters, for a given one of the compressed subsets, to facilitate forward propagation, includes deallocating client device memory for an additional one of the decompressed subsets utilized in an immediately preceding iteration of the forward propagation.
10 . The method of claim 8 , wherein the client ML model, received from the remote system, includes non-compressed model parameters that are also utilized to facilitate forward propagation.
11 . The method of claim 10 , wherein the non-compressed model parameters include a plurality of bias parameters, and the compressed model parameters include a plurality of weight parameters.
12 . The method of claim 8 , wherein the client ML model is utilized by the client device to facilitate automatic speech recognition (ASR) and an initial instance of the corresponding data is based on audio captured by the client device or another computing device.
13 . The method of claim 8 , wherein decompressing the compressed subset of parameters comprises:
applying a corresponding scaling factor and/or a corresponding biasing value to each parameter of the decompressed subset of parameters.
14 . A method implemented by one or more processors of a client device, the method comprising:
receiving, from a remote system, a client machine learning (ML) model in a compressed format;
for each of multiple compressed subsets of compressed model parameters of the client ML model, to facilitate forward propagation, and while all other of the compressed subsets of parameters of the client ML model remain compressed:
decompressing the compressed subset of parameters to generate a corresponding decompressed subset of parameters; and
processing corresponding data using the corresponding decompressed subset of parameters to generate a next iteration of the corresponding data and in furtherance of determining a predicted output from the forward propagation;
performing one or more actions based on the predicted output from the forward propagation.
15 . The method of claim 14 , wherein decompressing the compressed subset of parameters, for a given one of the compressed subsets, to facilitate forward propagation, includes deallocating client device memory for an additional one of the decompressed subsets utilized in an immediately preceding iteration of the forward propagation.
16 . The method of claim 14 , wherein the client ML model, received from the remote system, includes non-compressed model parameters that are also utilized to facilitate forward propagation.
17 . The method of claim 16 , wherein the non-compressed model parameters include a plurality of bias parameters, and the compressed model parameters include a plurality of weight parameters.
18 . The method of claim 14 , wherein the client ML model is utilized by the client device to facilitate automatic speech recognition (ASR) and an initial instance of the corresponding data is based on audio captured by the client device or another computing device.
19 . The method of claim 18 , wherein the one or more actions include:
rendering a transcription, of the audio data, determined based on the predicted output, and/or
controlling one or more applications based on the transcription determined based on the predicted output.
20 . The method of claim 14 , wherein decompressing the compressed subset of parameters comprises:
applying a corresponding scaling factor and/or a corresponding biasing value to each parameter of the decompressed subset of parameters.