IP Library › Granted Patent US 12,737,588
Granted Patent B2
US 12,737,588 · App. 17/244,781 · Granted Sep 15, 2026

Dynamic weight updates for neural networks

Inventors: Holger Reinhard Roth (Rockville, MD); Yingda Xia (Baltimore, MD); Daguang Xu (Potomac, MD); Andriy Myronenko (San Mateo, CA); Wenqi Li (London, GB); Dong Yang (Pocatello, ID)
Assignee: NVIDIA Corporation
G06N3/045G06F18/214G06N3/047H04L67/10H04L67/34
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,737,588
App. No.
17/244,781
Granted
Sep 15, 2026
Kind
B2
Abstract

Apparatuses, systems, and techniques to improve federated learning for neural networks. In at least one embodiment, a federated server dynamically selects neural network weights according to one or more learnable aggregation weights indicating a contribution from each of one or more edge devices or clients during federated training according to various characteristics of each edge device or client model and training data.

Claims (64)

1 . A computer system comprising:

one or more processors to:

determine a weighting for each of one or more sets of neural network weights stored by one or more federated learning clients based, at least in part, on respective degrees of one or more changes to the one or more sets of neural network weights resulting from one or more training operations completed by the one or more federated learning clients; and

apply the weightings to the one or more sets of neural network weights to generate one or more aggregated neural network weights to update a neural network.

2 . The computer system of claim 1 , wherein:

the one or more sets of neural network weights comprise numerical values; and

the one or more processors are to determine the weightings for one or more sets of the neural network weights using one or more aggregation values, the one or more aggregation values calculated as a result of one or more criteria applied to the one or more training operations.

3 . The computer system of claim 2 , wherein the one or more processors are to determine the weightings for one or more sets of neural network weights by selecting a first set of the neural network weights using a first aggregation value of the one or more aggregation values and selecting a second set of the neural network weights using a second aggregation value of the one or more aggregation values.

4 . The computer system of claim 2 , wherein:

the one or more processors update a neural network at least by applying the weightings to the one or more sets of neural network weights with one or more other sets of neural network weights from one or more other federated learning clients to generate the one or more aggregated neural network weights; and

the one or more aggregation values indicate how to determine the weightings for the one or more sets of neural network weights when aggregating with the one or more other sets of neural network weights.

5 . The computer system of claim 2 , wherein the one or more processors update a neural network based, at least in part, on the one or more aggregated neural network weights and distribute the updated neural network to the one or more federated learning clients.

6 . The computer system of claim 1 , wherein the one or more processors are to determine the weightings for the one or more sets of neural network weights based, at least in part, on one or more aggregation values, the one or more aggregation values determined, at least in part, as a result of one or more criteria applied to neural network training by the one or more federated learning clients to generate the one or more sets of neural network weights.

7 . The computer system of claim 2 , wherein the one or more criteria comprise at least a size of data usable by the one or more federated learning clients to generate the one or more sets of neural network weights.

8 . The computer system of claim 1 , wherein the one or more processors are to apply the weightings to the one or more sets of neural network weights in order to update a neural network to perform medical image segmentation.

9 . A method comprising:

determining a weighting for each of one or more sets of neural network weights stored by one or more federated learning clients based, at least in part, on respective degrees of one or more changes to the one or more sets of neural network weights resulting from one or more training operations completed by the one or more federated learning clients; and

applying the weightings to the one or more sets of neural network weights to generate one or more aggregated neural network weights to update a neural network.

10 . The method of claim 9 , wherein:

the one or more sets of neural network weights comprise one or more data values generated as a result of one or more training operations by the one or more federated learning clients.

11 . The method of claim 10 , further comprising distributing the updated neural network to the one or more federated learning clients.

12 . The method of claim 10 , further comprising determining one or more aggregation values based, at least in part, on applying one or more criteria by the one or more federated learning clients in the one or more training operations.

13 . The method of claim 9 , further comprising calculating one or more aggregation weights values based, at least in part, on applying one or more criteria to the one or more training operations by the one or more federated learning clients to generate the one or more sets of neural network weights, and determining the weightings for the one or more sets of neural network weights to be used to update the neural network.

14 . The method of claim 9 , further comprising determining the weightings for the one or more sets of neural network weights based on changes to data usable by the one or more federated learning clients to generate the one or more sets of neural network weights.

15 . The method of claim 9 , further comprising updating the neural network as a result of applying the weightings to the one or more sets of neural network weights to generate the one or more aggregated neural network weights and distributing the updated neural network to the one or more federated learning clients, the updated neural network comprising at least a subset of the one or more aggregated neural network weights.

16 . The method of claim 9 , further comprising applying the weightings to the one or more sets of neural network weights to update the neural network to perform medical image segmentation.

17 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:

determine a weighting for each of one or more sets of neural network weights stored by one or more federated learning clients based, at least in part, on respective degrees of one or more changes to the one or more sets of neural network weights resulting from one or more training operations completed by the one or more federated learning clients; and

apply the weighting to the one or more sets of neural network weights to generate one or more aggregated neural network weights to update a neural network.

18 . The non-transitory machine-readable medium of claim 17 , wherein the set of instructions further comprise instructions that, when executed by the one or more processors, cause the one or more processors to:

determine a first weighting for a first set of the neural network weights based, at least in part, on one or more sets of aggregation values;

determine a second weighting for a second set of the neural network weights based, at least in part, on the one or more sets of aggregation values; and

apply the first weighting and the second weighting to the first set and the second set to generate aggregated neural network weights and update the neural network using at least the aggregated neural network weights corresponding to the first set and the second set into a neural network.

19 . The non-transitory machine-readable medium of claim 18 , wherein the one or more sets of neural network weights comprise numerical values determined as a result of the one or more training operations performed by the one or more federated learning clients to generate the one or more sets of neural network weights.

20 . The non-transitory machine-readable medium of claim 18 , wherein the set of instructions further comprise instructions that, when executed by the one or more processors, cause the one or more processors to transfer the updated neural network to the one or more federated learning clients.

21 . The non-transitory machine-readable medium of claim 18 , wherein the set of instructions further comprise instructions that, when executed by the one or more processors, cause the one or more processors to compute the one or more sets of aggregation values based, at least in part, on applying one or more criteria to the one or more training operations by the one or more federated learning clients to generate the one or more sets of neural network weights.

22 . The non-transitory machine-readable medium of claim 18 , wherein the set of instructions further comprise instructions that, when executed by the one or more processors, cause the one or more processors to:

receive the one or more sets of aggregation values from the one or more federated learning clients;

calculate one or more other sets of aggregation values by applying weighted averaging to the one or more sets of aggregation values; and

determine the first weighting for the first set and the second weighting for the second set based, at least in part, on the one or more other sets of aggregation values.

23 . The non-transitory machine-readable medium of claim 21 , wherein the one or more criteria comprise at least a metric corresponding to a size of data usable by the one or more federated learning clients to generate the one or more sets of neural network weights.

24 . The non-transitory machine-readable medium of claim 21 , wherein the one or more criteria comprise at least a metric corresponding to one or more neural networks of the one or more federated learning clients used to generate the one or more sets of neural network weights.

25 . The non-transitory machine-readable medium of claim 17 , wherein the set of instructions further comprise instructions that, when executed by the one or more processors, cause the one or more processors to perform medical image segmentation using the updated neural network.

26 . One or more processors comprising:

circuitry to:

determine a weighting for each of one or more sets of neural network weights stored by one or more federated learning clients based, at least in part, on respective degrees of one or more changes to the one or more sets of neural network weights resulting from one or more training operations completed by the one or more federated learning clients; and

apply the weightings to the one or more sets of neural network weights to generate one or more aggregated neural network weights to update a neural network.

27 . The one or more processors of claim 26 , wherein:

the one or more sets of neural network weights comprise numerical values determined as a result of the one or more training operations being performed by the one or more federated learning clients; and

the circuitry is further to determine the weightings for the one or more sets of neural network weights using one or more aggregation values, the one or more aggregation values calculated as a result of one or more criteria applied to the one or more training operations performed by the one or more federated learning clients.

28 . The one or more processors of claim 27 , wherein the circuitry is further to update one or more neural networks using the one or more aggregated neural network weights, and further to cause the updated neural network to be distributed to the one or more federated learning clients.

29 . The one or more processors of claim 27 , wherein the one or more aggregation values indicate changes to data used by the one or more federated learning clients to generate the one or more sets of neural network weights according to the one or more criteria.

30 . The one or more processors of claim 27 , wherein the circuitry is to receive the one or more aggregation values from the one or more federated learning clients, the one or more aggregation values indicating how to determine the weightings for the one or more sets of neural network weights when applying the weightings to the one or more sets of neural network weights with one or more other sets of neural network weights to generate the one or more aggregated neural network weights.

31 . The one or more processors of claim 27 , wherein the one or more criteria are metrics computed based, at least in part, on data usable by the one or more federated learning clients to generate the one or more sets of neural network weights.

32 . The one or more processors of claim 26 , wherein the circuitry is further to apply the weightings to the one or more sets of neural network weights to generate the one or more aggregated neural network weights at least to update the neural network, the updated neural network distributed to each of the one or more federated learning clients to generate one or more sets of additional neural network weights.

33 . The one or more processors of claim 27 , wherein the one or more criteria comprise at least a size of data usable by the one or more federated learning clients to generate the one or more sets of neural network weights.

34 . The one or more processors of claim 27 , wherein the one or more criteria comprise at least a metric corresponding to one or more neural networks of the one or more federated learning clients used to generate the one or more sets of neural network weights.

35 . The one or more processors of claim 26 , wherein the one or more processors are to apply the weightings to the one or more sets of neural network weights to update a neural network to perform image classification.

36 . The computer system of claim 1 , wherein the one or more processors are to:

compute a validation loss for a first federated learning client of the one or more federated learning clients based on the one or more sets of neural network weights; and

determine the weighting for the first federated learning client based, at least in part, on the computed validation loss.

37 . The computer system of claim 1 , wherein the one or more processors are further to:

receive, from each of the one or more federated learning clients, both the one or more sets of neural network weights and one or more learnable aggregation values corresponding to that federated learning client; and

determine the weighting for each of the one or more federated learning clients based, at least in part, on the received one or more learnable aggregation values from that federated learning client.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 23, 2021
From: ROTH, HOLGER REINHARD; XIA, YINGDA; XU, DAGUANG; MYRONENKO, ANDRIY; LI, WENQI; YANG, DONG
To: NVIDIA CORPORATION
Reel/Frame 056964/0953 →
Continuity (1)
Related Publication 20220366220A1 · Nov 17, 2022
References Cited (87)
US 10825536B1 · Yang et al. · 2020 [cited by applicant]
US 12072954B1 · Li · 2024 [cited by examiner]
US 20190138534A1 · Bernat et al. · 2019 [cited by applicant]
US 20190156214A1 · Matveev et al. · 2019 [cited by applicant]
US 20190378008A1 · Markram et al. · 2019 [cited by applicant]
US 20200272899A1 · Dunne · 2020 [cited by examiner]
US 20200293887A1 · De Brouwer · 2020 [cited by examiner]
US 20200334810A1 · Accomazzi · 2020 [cited by examiner]
US 20200380306A1 · Hada et al. · 2020 [cited by applicant]
US 20210065002A1 · Samek · 2021 [cited by examiner]
US 20210081806A1 · Chai et al. · 2021 [cited by applicant]
US 20210287114A1 · Zhou · 2021 [cited by examiner]
US 20210374502A1 · Roth · 2021 [cited by examiner]
US 20220076133A1 · Yang · 2022 [cited by examiner]
US 20240303504A1 · Xu · 2024 [cited by examiner]
CN 112016669A · 2020 [cited by applicant]
EP 3699825A2 · 2020 [cited by applicant]
JP 2019212111A · 2019 [cited by applicant]
WO 2019230254A1 · 2019 [cited by applicant]
Office Action for Australian Application No. 2022202837, mailed Mar. 24, 2023, 8 pages. [cited by applicant]
United Kingdom Combined Search and Examination Report for Application No. GB2206263.2, mailed Dec. 12, 2022, 8 pages. [cited by applicant]
Zhao et al., “Federated Learning with Non-IID Data,” Jun. 2, 2018, 13 pages. [cited by applicant]
Zoph et al., “Neural Architecture Search with Reinforcement Learning,” ICLR, 2017, 16 pages. [cited by applicant]
Çiçek et al., “3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation,” International Conference on Medical Image Computing and Computer-Assisted Intervention, 2016, 8 pages. [cited by applicant]
Office Action for Australian Application No. 2022202837, mailed Mar. 20, 2024, 6 pages. [cited by applicant]
Munoz-Gonzalez et al., “Byzantine-Robust Federated Learning through Adaptive Model Averaging,” Sep. 11, 2019, 11 pages. [cited by applicant]
Office Action for United Kingdom Application No. GB2206263.2, mailed Jun. 10, 2024, 5 pages. [cited by applicant]
Yeganeh et al., “Inverse Distance Aggregation for Federated Learning with Non-IID Data,” Aug. 17, 2020, 11 pages. [cited by applicant]
Kagawa et al., “Accuracy Improvement Method for Collaborative Machine Learning with Individually Different Data and Consideration on Privacy,” IPSJ Symposium Computer Security Symposium, Oct. 2019, 9 pages. [cited by applicant]
Ma et al., “Design of Multi-Institutional Federated Learning Considering Medical Data Characteristics,” JCMI, Nov. 18, 2020, 10 pages. [cited by applicant]
Office Action for Japanese Application No. 2022-072004, mailed Aug. 30, 2024, 7 pages. [cited by applicant]
Andrychowicz et al., “Learning to Learn by Gradient Descent by Gradient Descent,” Advances in Neural Information Processing Systems, 2016, 9 pages. [cited by applicant]
Baker et al., “Designing Neural Network Architectures using Reinforcement Learning,” 2017, 16 pages. [cited by applicant]
Chang et al., “Synthetic Learning: Learn from Distributed Asynchronized Discriminator GAN without Sharing Medical Image Data,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, 11 … [cited by applicant]
Chen et al., “Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms,” Neural Information Processing Systems, 2020, 11 pages. [cited by applicant]
Chen et al., “DRNAS: Dirichlet Neural Architecture Search,” Oct. 16, 2020, 14 pages. [cited by applicant]
Chen et al., “Progressive Differentiable Architecture Search: Bridging the Depth GAP Between Search and Evaluation,” Proceedings of the IEEE International Conference on Computer Vision, 2019, 10 pages. [cited by applicant]
Cubuk et al., “Autoaugment: Learning Augmentation Policies from Data,” Oct. 9, 2018, 14 pages. [cited by applicant]
Cubuk et al., “Randaugment: Practical Automated Data Augmentation with a Reduced Search Space,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 14-19, 2020, 10 pages. [cited by applicant]
Dinh et al., “Personalized Federated Learning with Moreau Envelopes,” Neural Information Processing Systems, 2020, 12 pages. [cited by applicant]
Dou et al., “Domain Generalization via Model-Agnostic Learning of Semantic Features,” Advances in Neural Information Processing Systems, 2019, 12 pages. [cited by applicant]
Figurnov et al., “Implicit Reparameterization Gradients,” Advances in Neural Information Processing Systems, 2018, 12 pages. [cited by applicant]
Goodfellow et al., “Generative Adversarial Nets,” In Advances in Neural Information Processing Systems, Jun. 10, 2014, 9 pages. [cited by applicant]
Hanzely et al., “Lower Bounds and Optimal Algorithms for Personalized Federated Learning,” Neural Information Processing Systems, 2020, 12 pages. [cited by applicant]
Harmon et al., “Artificial Intelligence for the Detection of COVID-19 Pneumonia on Chest CT using Multinational Datasets,” Nature communications, 11(1): 2020, 7 pages. [cited by applicant]
IEEE, “IEEE Standard 754-2008 (Revision of IEEE Standard 754-1985): IEEE Standard for Floating-Point Arithmetic,” Aug. 29, 2008, 70 pages. [cited by applicant]
Jeong et al., “Communication Efficient on-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data,” Nov. 28, 2016, 6 pages. [cited by applicant]
Kairouz et al., “Advances and Open Problems in Federated Learning,” Dec. 10, 2019, 105 pages. [cited by applicant]
Kaissis et al., “Secure, Privacy-Preserving and Federated Machine Learning in Medical Imaging,” Nature Machine Intelligence, 2020, 7 pages. [cited by applicant]
Karimireddy et al., “SCAFFOLD: Stochastic Controlled Averaging for Federated Learning,” ICML, 2020, 12 pages. [cited by applicant]
Kingma et al., “Adam: A Method for Stochastic Optimization,” ICLR, 2015, 15 pages. [cited by applicant]
Landman et al., Miccai Multi-Atlas Labeling Beyond the Cranial Vault—Workshop and Challenge, MICCAI Multi-Atlas abeling Beyond Cranial Vault-Workshop Challenge, Apr. 15, 2015, 5 pages. [cited by applicant]
Li et al., Artificial Intelligence Distinguishes COVID-19 from Community Acquired Pneumonia on Chest CT, Radiology, 2020, 16 pages. [cited by applicant]
Li et al., “Federated Learning: Challenges, Methods, and Future Directions,” IEEE Signal Processing Magazine, 37(3):2020, 21 pages. [cited by applicant]
Li et al., “Federated Optimization in Heterogeneous Networks,” Dec. 14, 2018, 12 pages. [cited by applicant]
Li et al., “FedMD: Heterogenous Federated Learning via Model Distillation,” Oct. 8, 2019, 8 pages. [cited by applicant]
Li et al., “Multi-Site fMRI Analysis Using Privacy-Preserving Federated Learning and Domain Adaptation: ABIDE Results,” Dec. 6, 2020, 16 pages. [cited by applicant]
Li et al., “On the Convergence of FedAvg on Non-IID Data,” Oct. 8, 2019, 26 pages. [cited by applicant]
Li et al., “Privacy-Preserving Federated Brain Tumour Segmentation,” International Workshop on Machine Learning in Medical Imaging, 2019, 8 pages. [cited by applicant]
Liang et al., “Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization,” Neural Information Processing Systems, 2020, 13 pages. [cited by applicant]
Liu et al., “DARTS: Differentiable Architecture Search,” Apr. 23, 2019, 13 pages. [cited by applicant]
Liu et al., “Msnet: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data,” IEEE Transactions on Medical Imaging, 2020, 12 pages. [cited by applicant]
Lowe et al., “Precision Medicine in Pancreatic Disease-Knowledge GAPs and Research Opportunities: Summary of a National Institute of Diabetes and Digestive and Kidney Diseases Workshop,” Pancreas, 48(10): 2019, 20 pages. [cited by applicant]
Mcmahan et al., “Communication Efficient Learning of Deep Networks from Decentralized Data,” Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, 2017, 10 pages. [cited by applicant]
Milletari et al., “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” 2016 Fourth International Conference on 3D Vision (3DV), Oct. 25, 2016, 11 pages. [cited by applicant]
Mohri et al., “Agnostic Federated Learning,” International Conference on Machine Learning, 2019, 11 pages. [cited by applicant]
Real et al., “Regularized Evolution for Image Classifier Architecture Search,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, 10 pages. [cited by applicant]
Rieke et al., “The Future of Digital Health with Federated Learning,” NPJ Digital Medicine, 2020, 7 pages. [cited by applicant]
Ronneberger et al., “U-net: Convolutional networks for biomedical image segmentation,” International Conference on Medical Image Computing and Computer-Assisted Intervention, Oct. 5, 2015, 8 pages. [cited by applicant]
Roth et al., “Deeporgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation,” International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, Jun. 23, 2015, 12 … [cited by applicant]
Roth et al., “Federated Learning for Breast Density Classification: A Real-World Implementation,” Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, 2020, 11 pages. [cited by applicant]
Sattler et al., “Robust and Communication-Efficient Federated Learning from Non-IID Data,” IEEE Transactions on Neural Networks and Learning Systems, 2019, 17 pages. [cited by applicant]
Sheller et al., “Federated Learning in Medicine: Facilitating Multi-Institutional Collaborations without Sharing Patient Data,” Scientific reports, 10(1): 2020, 13 pages. [cited by applicant]
Shi et al., “Review of Artificial Intelligence Techniques in Imaging Data Acquisition, Segmentation and Diagnosis for COVID-19,” IEEE Reviews in Biomedical Engineering, Apr. 6, 2020, 11 pages. [cited by applicant]
Simpson et al.: A Large Annotated Medical Image Dataset for the Development and Evaluation of Segmentation Algorithms, Feb. 25, 2019, 15 pages. [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201609, issued Jan… [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201806, issued Jan… [cited by applicant]
Wang et al., “Federated Learning with Matched Averaging,” ICLR, 2020, 16 pages. [cited by applicant]
Xia et al., “Uncertainty-Aware Multi-View Co-Training for Semi-Supervised Medical Image Segmentation and Domain Adaptation,” Medical Image Analysis, 2020, 19 pages. [cited by applicant]
Xie et al., “Genetic CNN,” ICCV, 2017, 10 pages. [cited by applicant]
Yang et al., “Federated Machine Learning: Concept and Applications,” ACM Transactions on Intelligent Systems and Technology 10(2): 2019, 19 pages. [cited by applicant]
Examination Report for United Kingdom Application No. GB2206263.2, mailed Jun. 17, 2025, 6 pages. [cited by applicant]
Sun et al., “Adaptive Federated Learning and Digital Twin for Industrial Internet of Things,” 2020, 10 pages. [cited by applicant]
Examination Report for Australian Application No. 2024201907, mailed Dec. 15, 2025, 9 pages. [cited by applicant]
Examination Report for United Kingdom Application No. GB2206263.2, mailed Sep. 2, 2025, 6 pages. [cited by applicant]
Office Action for Chinese Application No. 202210411396.3, mailed Dec. 11, 2025, 24 pages. [cited by applicant]
Office Action for Chinese Application No. 202210411396.3, mailed May 12, 2026, 21 pages. [cited by applicant]