IP Library › Granted Patent US 12,450,489
Granted Patent B2
US 12,450,489 · App. 17/586,178 · Granted Oct 21, 2025

Method, system and apparatus for federated learning

Inventors: Stefanos Laskaridis (Middlesex, GB); Samuel Horvath (Middlesex, GB); Mario Almeida (Middlesex, GB); Ilias Leontiadis (Middlesex, GB); Stylianos I. Venieris (Middlesex, GB)
Assignee: Samsung Electronics Co., Ltd.
G06N3/082G06N3/045
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,450,489
App. No.
17/586,178
Granted
Oct 21, 2025
Kind
B2
Abstract

Broadly speaking, the present techniques generally relates to methods, systems and apparatuses for training a machine learning (ML) model using federated learning. In particular, a method for training a machine learning (ML) model using federated learning performed by a plurality of client devices, the method comprising determining a computation capability of each client device, associating each client device with a value defining how much of each neural network layer of the ML model is to be included in a submodel to be trained by the each client device, based on the determined computation capability and generating a submodel of the ML model by using the value associated with the each client device to perform ordered pruning of at least one neural network layer of the ML model, is provided.

Claims (56)

1. A method performed by a central computing device for training a machine learning (ML) model comprising a neural network using federated learning performed by a plurality of client devices, the method comprising:

determining a computation capability of each client device of the plurality of client devices;

associating the each client device with a dropout rate defining how much of each neural network layer of the ML model is to be included in a target submodel to be trained by the each client device, based on the determined computation capability;

generating different sized submodels of the ML model that are nested within the ML model such a largest submodel of the different sized submodels is included in the ML model and each of an increasingly smaller in size submodel of the different sized submodels is included in a next larger in size submodel of the different sized submodels, the different sized submodels being generated by using the dropout rate associated with the each client device to perform pruning of at least one neuron of at least one neural network layer of the ML model according to a predefined ordering such that the at least one neuron that is pruned is not used by any submodel of the different sized submodels for which the at least one neuron is pruned; and

distributing, during each federated learning training round, at least two submodels of the different sized submodels to at least one client device of the plurality of client devices based on the dropout rate associated with the at least one client device, the at least two submodels including, a submodel and at least one submodel nested within the submodel, from among the different sized submodels,

wherein the generating of the different sized submodels of the ML model that are nested within the ML model comprises:

using the dropout rate associated with the each client device to perform the ordered pruning of one neural network layer of the ML model; and

using at least one further dropout rate to perform the pruning of at least one further neural network layer of the ML model.

2. The method as claimed in claim 1 , wherein the associating of the each client device with the dropout rate comprises associating the each client device with the dropout rate from a distribution of discrete dropout rates.

3. The method as claimed in claim 2 , wherein the distribution of the discrete dropout rates is a uniform distribution.

4. The method as claimed in claim 2 , wherein the distribution of the discrete dropout rates is a non-uniform distribution which reflects a distribution of capabilities of the plurality of client devices.

5. The method as claimed in claim 2 , wherein a number of the discrete dropout rates in the distribution of the discrete dropout rates depends on at least one of:

a number of device computation capability levels;

a size or width of neural network layers; or

a number of the device computation capability levels included in the ML model.

6. The method as claimed in claim 1 , wherein the generating of the different sized submodels of the ML model that are nested within the ML model comprises using the dropout rate associated with the each client device to perform ordered pruning of the each neural network layer of the ML model.

7. The method as claimed in claim 1 , wherein the distributing, during the each federated learning training round, of the at least two submodels to the at least one client device comprises broadcasting the submodel, which is generated using the dropout rate, to the at least one client device associated with the dropout rate, for local training by the at least one client device.

8. The method as claimed in claim 1 , further comprising:

receiving, during the each federated learning training round, gradients of a supported updated submodel from the at least one client device, where each gradient corresponds to changes in weights of the submodel.

9. The method as claimed in claim 8 , further comprising:

aggregating, using the received gradients, the changes in weights of the submodel received from at least the at least one client device; and

updating the ML model.

10. The method as claimed in claim 9 , further comprising:

repeating, the generating of the different sized submodels of the ML model that are nested within the ML model and the distributing of the at least two submodels to the at least one client device, using the updated ML model.

11. The method as claimed in claim 1 , wherein the ordering is predefined based on importance of the at least one neuron.

12. One or more non-transitory computer-readable storage media storing instructions that, when individually or collectively executed by at least one processor of a central computing device for training a machine learning (ML) model comprising a neural network using federated learning performed by a plurality of client devices, cause the central computing device to perform operations, the operations comprising:

determining a computation capability of each client device of the plurality of client devices;

associating the each client device with a dropout rate defining how much of each neural network layer of the ML model is to be included in a target submodel to be trained by the each client device, based on the determined computation capability;

generating different sized submodels of the ML model that are nested within the ML model such a largest submodel of the different sized submodels is included in the ML model and each of an increasingly smaller in size submodel of the different sized submodels is included in a next larger in size submodel of the different sized submodels, the different sized submodels being generated by using the dropout rate associated with the each client device to perform pruning of at least one neuron of at least one neural network layer of the ML model according to a predefined ordering such that the at least one neuron that is pruned is not used by any submodel of the different sized submodels for which the at least one neuron is pruned; and

distributing, during each federated learning training round, at least two submodels of the different sized submodels to at least one client device of the plurality of client devices based on the dropout rate associated with the at least one client device, the at least two submodels including, a submodel and at least one submodel nested within the submodel, from among the different sized submodels,

wherein the generating of the different sized submodels of the ML model that are nested within the ML model comprises:

using the dropout rate associated with the each client device to perform the ordered pruning of one neural network layer of the ML model; and

using at least one further dropout rate to perform the pruning of at least one further neural network layer of the ML model.

13. The one or more non-transitory computer-readable storage media of claim 12 , wherein the ordering is predefined based on importance of the at least one neuron.

14. A system for training a machine learning (ML) model comprising a neural network using federated learning performed by a plurality of client devices, the system comprising:

the plurality of client devices; and

a central computing device comprising:

memory storing instructions, and

at least one processor communicatively coupled to the memory,

wherein the instructions, when executed by the at least one processor individually or collectively, cause the central computing device to:

determine a capability of each client device of the plurality of client devices,

associate the each client device with a dropout rate defining how much of each neural network layer of the ML model is to be included in a target submodel implemented by the each client device, based on the determined computation capability,

generate a different sized submodels of the ML model that are nested within the ML model such a largest submodel of the different sized submodels is included in the ML model and each of an increasingly smaller in size submodel of the different sized submodels is included in a next larger in size submodel of the different sized submodels, the different sized submodels being generated by using the dropout rate associated with the each client device to perform pruning of at least one neuron of at least one neural network layer of the ML model according to a predefined ordering such that the at least one neuron that is pruned is not used by any submodel of the different sized submodels for which the at least one neuron is pruned, and

distribute, during each federated learning training round, at least two submodels of the different sized submodels to at least one client device of the plurality of client devices based on the dropout rate associated with the at least one client device, the at least two submodels including, a submodel and at least one submodel nested within the submodel, from among the different sized submodels, and

wherein to generate the different sized submodels of the ML model that are nested within the ML model, the instructions, when executed by the at least one processor individually or collectively, further cause the central computing device to:

use the dropout rate associated with the each client device to perform the ordered pruning of one neural network layer of the ML model; and

use at least one further dropout rate to perform the pruning of at least one further neural network layer of the ML model.

15. The system as claimed in claim 14 , wherein the instructions, when executed by the at least one processor individually or collectively, further cause the central computing device to:

associate the each client device with the dropout rate from a distribution of discrete dropout rates.

16. The system as claimed in claim 15 , wherein the distribution of the discrete dropout rates is a uniform distribution.

17. The system as claimed in claim 15 , wherein the distribution of the discrete dropout rates is a non-uniform distribution which reflects a distribution of capabilities of the plurality of client devices.

18. The system as claimed in claim 15 , wherein a number of the discrete dropout rates in the distribution of the discrete dropout rates depends on at least one of:

a number of device computation capability levels,

a size or width of neural network layers, or

a number of the device computation capability levels included in the ML model.

19. The system as claimed in claim 14 , wherein the ordering is predefined based on importance of the at least one neuron.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 27, 2022
From: LASKARIDIS, STEFANOS; VENIERIS, STYLIANOS I.
To: SAMSUNG ELECTRONICS CO., LTD.
Reel/Frame 058795/0906 →
Priority Claims (2)
GR 20210100067 · Feb 2, 2021 · national
EP 21195060 · Sep 6, 2021 · regional
Continuity (2)
Continuation PCTKR2022000938 · Jan 18, 2022
Related Publication 20220245459A1 · Aug 4, 2022
References Cited (49)
US 10657461B2 · McMahan et al. · 2020 [cited by applicant]
US 20150100530A1 · Mnih · 2015 [cited by examiner]
US 20170076198A1 · Jin et al. · 2017 [cited by applicant]
US 20180349769A1 · Mankovskii et al. · 2018 [cited by applicant]
US 20200245009A1 · Saini et al. · 2020 [cited by applicant]
US 20200364608A1 · Anwar et al. · 2020 [cited by applicant]
CN 107871160A · 2018 [cited by applicant]
CN 111865815A · 2020 [cited by applicant]
KR 1020200133484 · 2020 [cited by applicant]
WO WO2022028666A1 · 2022 [cited by examiner]
WO 2020222407A1 · 2022 [cited by applicant]
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz. “Pruning Convolutional Neural Networks for Resource Efficient Inference”. arXiv:1611.06440. (Year: 2017). [cited by examiner]
Song Han, Jeff Pool, John Tran, William J. Dally. “Learning both Weights and Connections for Efficient Neural Networks”. arXiv: 1506.02626. (Year: 2015). [cited by examiner]
Duchi et al., Privacy Aware Learning, Advances in Neural Information Processing Systems 25_2012. [cited by applicant]
Alistarh et al., QSGD: Communication-efficient SGD via gradient quantization and encoding. Advances in Neural Information Processing Systems 2017. [cited by applicant]
Wen et al., TernGrad: Ternary gradients to reduce communication in distributed deep learning. Advances in Neural Information Processing Systems, Dec. 29, 2017. [cited by applicant]
Jiang et al., Model Pruning Enables Efficient Federated Learning on Edge Devices, Oct. 23, 2020. [cited by applicant]
Caldas et al., Expanding the Reach of Federated Learning by Reducing Client Resource Requirements, Jan. 8, 2019. [cited by applicant]
Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(56): Journal of Machine Learning Research 15 (2014) 1929-1958. [cited by applicant]
Mcmahan et al., Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, Feb. 28, 2017. [cited by applicant]
Li et al., Federated Optimization in Heterogeneous Networks. MLSys, Apr. 21, 2020. [cited by applicant]
Wang et al., Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. NeurIPS, 2020. [cited by applicant]
Karimireddy et al., SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. International Conference on Machine Learning, Apr. 9, 2021. [cited by applicant]
Abadi et al., Deep Learning with Differential Privacy, Oct. 24, 2016. [cited by applicant]
Zagoruyko et al., Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer. CoRR, Dec. 12, 2016. [cited by applicant]
Wang et al., Adaptive Federated Learning in Resource Constrained Edge Computing Systems, Feb. 17, 2019. [cited by applicant]
Konecny et al., Federated Learning: Strategies for Improving Communication Efficiency. pp. 1.10, Oct. 30, 2017. [cited by applicant]
Wang et al., ATOMO: Communication-efficient learning via atomic sparsification. Advances in Neural Information Processing Systems, Nov. 8, 2018. [cited by applicant]
Amiri et al., Federated Learning With Quantized Global Model Updates, Oct. 7, 2020. [cited by applicant]
Han et al., Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach. ICDCS, Mar. 20, 2020. [cited by applicant]
Smith et al., Federated multi-task learning. In Advances in neural information processing systems, 2017. [cited by applicant]
Han et al., Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. ICLR, Feb. 15, 2016. [cited by applicant]
Zhou et al., DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients, Feb. 2, 2018. [cited by applicant]
Rajagopal et al., Multi-Precision Policy Enforced Training (MuPPET): A Precision-Switching Strategy for Quantised Fixed-Point Training of CNNs, Jun. 16, 2020. [cited by applicant]
Luo et al., Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International conference on computer vision, Jul. 20, 2017. [cited by applicant]
Yu et al., Scalpel: Customizing DNN pruning to the underlying hardware parallelism. ACM SIGARCH Computer Architecture News, 2017. [cited by applicant]
Wang et al., Beyond filters: Compact feature map for portable deep model. In International Conference on Machine Learning, 2017. [cited by applicant]
Molchanov et al., Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 25, 2019. [cited by applicant]
Rippel et al., Learning ordered representations with nested dropout. 31st International Conference on Machine Learning. 2014. [cited by applicant]
Howard et al., MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv, Apr. 17, 2017. [cited by applicant]
Sandler et al., MobileNetV2: Inverted Residuals and Linear Bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Apr. 2, 2018. [cited by applicant]
Zhang et al., ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Dec. 7, 2017. [cited by applicant]
Hinton et al., Distilling the Knowledge in a Neural Network. In NIPS 2014 Deep Learning Workshop, Mar. 9, 2015. [cited by applicant]
Diao et al., XP 055890230 (Computation and Communication Efficient Federated Learning for Heteroge Neous Clients), Oct. 3, 2020. [cited by applicant]
Bouacida et al., XP 081808992 (Adaptive Federated Dropout Improving Communication Efficiency and Generalization for Federated Learning), Nov. 8, 2020. [cited by applicant]
European Search Report dated Mar. 1, 2022, issued in European Patent Application No. 21195060.5. [cited by applicant]
International Search Report with Written Opinion dated May 2, 2022, issued in International Patent Application No. PCT/KR2022/000938. [cited by applicant]
European Communication pursuant to Article 94(3) EPC dated May 19, 2025; European Appln. No. 21 195 060.5-1207. [cited by applicant]
Chinese Office Action with English translation dated Jul. 1, 2025; Chinese Appln. No. 202280013009.9. [cited by applicant]