IP Library › Granted Patent US 12,190,247
Granted Patent B2
US 12,190,247 · App. 18/449,662 · Granted Jan 7, 2025

Systems and methods for distributed training of deep learning models

Inventor: David Moloney (Dublin, IE)
Assignee: Intel Corporation
G06N3/084G06F18/214G06N3/045G06N3/08G06V10/454G06V10/764G06V10/82G06V10/95G06V10/96H04L67/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,190,247
App. No.
18/449,662
Granted
Jan 7, 2025
Kind
B2
Abstract

Systems and methods for distributed training of deep learning models are disclosed. An example local device to train deep learning models includes a reference generator to label input data received at the local device to generate training data, a trainer to train a local deep learning model and to transmit the local deep learning model to a server that is to receive a plurality of local deep learning models from a plurality of local devices, the server to determine a set of weights for a global deep learning model, and an updater to update the local deep learning model based on the set of weights received from the server.

Claims (46)

1. A non-transitory computer readable medium comprising instructions that, when executed, cause a machine to:

obtain first weights associated with a first machine learning model trained by a first computing device;

obtain second weights associated with a second machine learning model trained by a second computing device;

aggregate the first weights associated with the first machine learning model and the second weights associated with the second machine learning model;

update a global machine learning model with the aggregated first weights and the aggregated second weights; and

share the updated global machine learning model to the first computing device and the second computing device, the first computing device to generate an updated first machine learning model based on the updated global machine learning model, the second computing device to generate an updated second machine learning model based on the updated global machine learning model.

2. The non-transitory computer readable medium of claim 1 , wherein the first computing device is local to a first location and the second computing device is local to a second location, the first location different than the second location.

3. The non-transitory computer readable medium of claim 1 , wherein the first machine learning model is obtained by the first computing device from a central server device where the global machine learning model is stored.

4. The non-transitory computer readable medium of claim 3 , wherein the second machine learning model is obtained by the second computing device from the central server device where the global machine learning model is stored.

5. The non-transitory computer readable medium of claim 1 , wherein the first computing device is to calculate the first weights associated with the first machine learning model based on local data at the first computing device.

6. The non-transitory computer readable medium of claim 5 , wherein the second computing device is to calculate the second weights associated with the second machine learning model based on local data at the second computing device.

7. A non-transitory computer readable medium comprising instructions that, when executed, cause a machine to:

obtain a global model from a server;

label input data based on a reference system of the global model;

train a machine learning model using the global model based on a difference between a label for the input data indicated from the reference system of the global model and an output of the machine learning model based on the input data;

calculate weights associated with the trained machine learning model; and

transmit the weights associated with the trained machine learning model to the server, the server to update the global model with the weights associated with the trained machine learning model.

8. The non-transitory computer readable medium of claim 7 , further including instructions to cause the machine to calculate the weights associated with the trained machine learning model based on local data.

9. The non-transitory computer readable medium of claim 7 , wherein the calculated weights are first weights, the trained machine learning model is a first trained machine learning model, and the server is to update the global model using the first weights associated with the first trained machine learning model and second weights associated with a second trained machine learning model.

10. The non-transitory computer readable medium of claim 9 , wherein the machine learning model is a first machine learning model, further including instructions to cause the machine to train the first machine learning model with the global model updated with the first weights and the second weights.

11. The non-transitory computer readable medium of claim 9 , wherein the machine is a first machine, the second weights associated with the second trained machine learning model are based on local data of a second machine, and the first machine is different than the second machine.

12. The non-transitory computer readable medium of claim 7 , wherein the reference system is a target system to be modelled by the machine learning model.

13. The non-transitory computer readable medium of claim 12 , wherein the reference system includes a transfer function to be learned by the machine learning model.

14. A system comprising:

a first computing device to:

obtain a global machine learning model;

train a first machine learning model using the global machine learning model; and

calculate first weights associated with the first trained machine learning model;

a second computing device to:

obtain a global machine learning model;

train a second machine learning model using the global machine learning model; and

calculate second weights associated with the second trained machine learning model; and

a server to:

obtain the first weights associated with the first trained machine learning model from the first computing device;

obtain the second weights associated with the second trained machine learning model from the second computing device;

aggregate the first weights associated with the first machine learning model and the second weights associated with the second machine learning model;

update the global machine learning model with the aggregated first weights and second weights; and

share the updated global machine learning model to the first computing device and the second computing device, the first computing device to generate an updated first machine learning model based on the updated global machine learning model, the second computing device to generate an updated second machine learning model based on the updated global machine learning model.

15. The system of claim 14 , wherein the first computing device is local to a first location and the second computing device is local to a second location, the first location different than the second location.

16. The system of claim 14 , wherein the first computing device is to calculate the first weights associated with the first trained machine learning model based on local data at the first computing device.

17. The system of claim 14 , wherein the second computing device is to calculate the second weights associated with the second trained machine learning model based on local data at the second computing device.

18. The system of claim 14 , wherein:

the first computing device is to transmit the first weights associated with the first trained machine learning model to the server; and

the second computing device is to transmit the second weights associated with the second trained machine learning model to the server.

19. The system of claim 14 , wherein the first machine learning model is obtained by the first computing device from a central server device where the global machine learning model is stored.

20. The system of claim 19 , wherein the second machine learning model is obtained by the second computing device from the central server device where the global machine learning model is stored.

Continuity (4)
Continuation 18145004 · Dec 21, 2022
Continuation 16326361
Provisional Application 62377094 · Aug 19, 2016
Related Publication 20240013056A1 · Jan 11, 2024
References Cited (69)
US 8311967B1 · Lin et al. · 2012 [cited by applicant]
US 8521664B1 · Lin et al. · 2013 [cited by applicant]
US 9639777B1 · Moloney et al. · 2017 [cited by applicant]
US 10402469B2 · McMahan et al. · 2019 [cited by applicant]
US 11170309B1 · Stefani et al. · 2021 [cited by applicant]
US 11580380B2 · Moloney · 2023 [cited by examiner]
US 11595274B1 · Nagaraju · 2023 [cited by examiner]
US 11610156B1 · Nagaraju · 2023 [cited by examiner]
US 11620549B2 · Sanchez · 2023 [cited by examiner]
US 11720955B2 · Chuah · 2023 [cited by examiner]
US 11769059B2 · Moloney · 2023 [cited by examiner]
US 20070089174A1 · Bader · 2007 [cited by examiner]
US 20140067738A1 · Kingsbury · 2014 [cited by applicant]
US 20140279741A1 · Sow et al. · 2014 [cited by applicant]
US 20140349269A1 · Canoy et al. · 2014 [cited by applicant]
US 20140351374A1 · Canoy · 2014 [cited by applicant]
US 20150242760A1 · Miao · 2015 [cited by examiner]
US 20150254555A1 · Williams, Jr. et al. · 2015 [cited by applicant]
US 20150324686A1 · Julian et al. · 2015 [cited by applicant]
US 20150324690A1 · Chilimbi et al. · 2015 [cited by applicant]
US 20150356461A1 · Vinyals et al. · 2015 [cited by applicant]
US 20150363634A1 · Yin et al. · 2015 [cited by applicant]
US 20160063393A1 · Ramage et al. · 2016 [cited by applicant]
US 20160379115A1 · Burger et al. · 2016 [cited by applicant]
US 20170109322A1 · McMahan · 2017 [cited by examiner]
US 20170147920A1 · Huo et al. · 2017 [cited by applicant]
US 20180053042A1 · Philbin et al. · 2018 [cited by applicant]
US 20180210874A1 · Fuxman et al. · 2018 [cited by applicant]
US 20180232528A1 · Williamson · 2018 [cited by examiner]
US 20180350050A1 · Finlayson et al. · 2018 [cited by applicant]
US 20190180195A1 · Terry et al. · 2019 [cited by applicant]
US 20190180196A1 · Terry et al. · 2019 [cited by applicant]
US 20200044801A1 · Wang et al. · 2020 [cited by applicant]
CN 104992430A · 2015 [cited by applicant]
CN 105160397A · 2015 [cited by applicant]
CN 105378762A · 2016 [cited by applicant]
CN 105447866A · 2016 [cited by applicant]
CN 105488515A · 2016 [cited by applicant]
CN 105531725A · 2016 [cited by applicant]
CN 105630882A · 2016 [cited by applicant]
CN 105678332A · 2016 [cited by applicant]
GB 201520932 · 2016 [cited by applicant]
JP 2013073301A · 2013 [cited by applicant]
KR 20160013107A · 2016 [cited by applicant]
KR 20160034814A · 2016 [cited by applicant]
WO 2015126858A1 · 2015 [cited by applicant]
Emilliano Miluzzo et al.“Darwin phones,” Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services, Jun. 15, 2012, 16 pages. [cited by applicant]
Emilliano Miluzzo et al. “Vision: mClouds-computing on clouds of mobile devices,” Proceedings of the 3rd ACM Workshop on Mobile Cloud Computing and Services (MCS'12), Jun. 25, 2012, 5 pages. [cited by applicant]
European Patent Office, “Extended European Search Report,” issued in connection with European patent Application No. 23209185.0-1203, dated Feb. 15, 2024, 8 pages. [cited by applicant]
International Searching Authority, “International Search Report,” issued in connection with PCT No. PCT/IB2021/055024, Dec. 12, 2017, 3 pages. [cited by applicant]
International Searching Authority, “Written Opinion of the International Searching Authority,” issued in connection with International Patent Application No. PCT/IB2017/055024, dated Dec. 12, 2017, 5 pages. [cited by applicant]
International Searching Authority, “International Preliminary Report on Patentability,” issued in connection with PCT No. PCT/IB2021/055024, Feb. 19, 2019, 6 pages. [cited by applicant]
European Patent Office, “Communication pursuant to Article 94(3) EPC,” issued in connection with European patent application No. 17771869.9, Feb. 7, 2020, 5 pages. [cited by applicant]
European Patent Office, “Communication Pursuant to Article 94(3) EPC,” issued in connection with application No. 17771869.9 dated May 14, 2021, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Action” issued in U.S. Appl. No. 16/326,361 on Jun. 27, 2022 (18 pages). [cited by applicant]
The State Intellectual Property Office If the People's Republic of China, “The First Office Action,” issued in connection with Application No. 201780064431.6, dated Sep. 5, 2022, 4 Pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance” issued in U.S. Appl. No. 16/326,361 on Oct. 13, 2022 (7 pages). [cited by applicant]
Korean Intellectual Property Office, “Request for the Submission of an Opinion,” issued in connection with Korean Patent Application No. 20197007900, dated Oct. 20, 2022, 18 pages. (English translation included). [cited by applicant]
Korean Intellectual Property Office, “Notice of Preliminary Rejection,” issued in connection with Korean Patent Application No. 10-2019-7007900 issued on Oct. 20, 2022, 19 pages (English language machine translation inc… [cited by applicant]
Korean Intellectual Property Office, “Written Decision on Registration,” issued in connection with Korean Patent Application No. 20197007900, dated Apr. 19, 2023, 5 pages. (English translation included). [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/145,004, mailed on May 12, 2023, 8 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 18/145,004, mailed on May 19, 2023, 1 page. [cited by applicant]
The State Intellectual Property Office of People's Republic of China, “Second Office Action,” issued in connection with Chinese Patent Application No. 201780064431.6, dated May 20, 2023, 20 pages. [English translation I… [cited by applicant]
The State Intellectual Property Office of People's Republic of China, “Supplementary search,” issued in connection with Chinese Patent Application No. 201780064431.6, dated May 20, 2023, 2 pages. [cited by applicant]
The State Intellectual Property Office of People's Republic of China, “Rejection decision,” issued in connection with Chinese Patent Application No. 201780064431.6, dated Sep. 29, 2023, 24 pages. [Machine English transl… [cited by applicant]
European Patent Office, “Summons to attend Oral Proceedings pursuant to Rule 115(1) EPC,” issued in connection with European patent Application No. 17 771 869.9-1203, on Oct. 18, 2023, 7 pages. [cited by applicant]
The State Intellectual Property Office of People's Republic of China, “Correction notice,” issued in connection with Chinese Patent Application No. 202311499504.8, dated Dec. 11, 2023, 1 page. [cited by applicant]
European Patent Office, “Brief Communication,” issued in connection with European Patent Application No. 17 771 869.9-1203, mailed on Apr. 26, 2024, 3 pages. [cited by applicant]
European Patent Office, “Communication under Rule 71(3) EPC,” issued in connection with European Application No. 17771869.9, dated Jun. 28, 2024, 11 pages. [cited by applicant]