IP Library › Granted Patent US 12,705,500
Granted Patent B2
US 12,705,500 · App. 18/365,734 · Granted Aug 11, 2026

Communication efficient federated learning

Inventors: Hugh Brendan McMahan (Seattle, WA); Dave Morris Bacon (Seattle, WA); Jakub Konecny (Edinburgh, GB); Xinnan Yu (Forest Hills, NY)
Assignee: GOOGLE LLC
G06N3/098G06F17/16G06F17/18G06N7/01G06N20/00G06F7/582
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,705,500
App. No.
18/365,734
Filed
Aug 4, 2023
Granted
Aug 11, 2026
Kind
B2
Art Unit
2124
USPC
706/10
Abstract

The present disclosure provides efficient communication techniques for transmission of model updates within a machine learning framework, such as, for example, a federated learning framework in which a high-quality centralized model is trained on training data distributed overt a large number of clients each with unreliable network connections and low computational power. In an example federated learning setting, in each of a plurality of rounds, each client independently updates the model based on its local data and communicates the updated model back to the server, where all the client-side updates are used to update a global model. The present disclosure provides systems and methods that reduce communication costs. In particular, the present disclosure provides at least: structured update approaches in which the model update is restricted to be small and sketched update approaches in which the model update is compressed before sending to the server.

Claims (44)

1 . A computer-implemented method for communication efficient machine learning, the method comprising:

obtaining, by a client computing device, global values for a set of parameters of a machine-learned model, wherein the machine-learned model is stored on the client computing device;

training, by the client computing device, the machine-learned model stored on the client computing device based at least in part on a local dataset stored on the client computing device to obtain an update matrix that is descriptive of updated values for the set of parameters of the machine-learned model, wherein the local dataset comprises data descriptive of an input received by the client computing device from a user of the client computing device, wherein training, by the client computing device, the machine-learned model based at least in part on the local dataset comprises:

training, by the client computing device, the machine-learned model based at least in part on the local dataset such that updated values are determined only for a pre-selected portion of the set of parameters, and wherein the update matrix is descriptive of only the updated values for the pre-selected portion of the set of parameters; and

communicating, by the client computing device, information descriptive of the update matrix to a server computing device.

2 . The computer-implemented method of claim 1 , wherein the update matrix is restricted to be a low-rank matrix.

3 . The computer-implemented method of claim 1 , wherein:

training, by the client computing device, the machine-learned model based at least in part on the local dataset to obtain the update matrix comprises:

defining, by the client computing device, the update matrix as a product of a first matrix and a second matrix, wherein the first matrix comprises fixed values and the second matrix comprises optimizable variables; and

training, by the client computing device, machine-learned model based at least in part on the local dataset to obtain the second matrix; and

wherein communicating, by the client computing device, information descriptive of the update matrix to the server computing device comprises communicating, by the client computing device, information descriptive of the second matrix to the server computing device.

4 . The computer-implemented method of claim 3 , further comprising, prior to training, by the client computing device, the machine-learned model:

generating, by the client computing device, the first matrix based at least in part on a seed and a pseudo-random number generator, wherein both the client computing device and the server computing device have knowledge of the seed such that the first matrix is reproducible by the server computing device.

5 . The computer-implemented method of claim 1 , wherein the update matrix is restricted to be a sparse matrix.

6 . The computer-implemented method of claim 1 , further comprising, prior to training, by the client computing device, the machine-learned model:

masking, by the client computing device, the pre-selected portion of the set of parameters from the set of parameters via a parameter mask.

7 . The computer-implemented method of claim 6 , wherein masking, by the client computing device, the pre-selected portion of the set of parameters is based at least in part on a seed and a pseudo-random number generator, wherein both the client computing device and the server computing device have knowledge of the seed such that the parameter mask is reproducible by the server computing device.

8 . The computer-implemented method of claim 1 , wherein the update matrix describes the updated values for the set of parameters or respective differences between the updated values and the global values.

9 . A client computing device, comprising:

at least one processor; and

at least one non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the client computing device to perform operations, the operations comprising:

obtaining global values for a set of parameters of a machine-learned model, wherein the machine-learned model is stored on the client computing device;

training the machine-learned model stored on the client computing device based at least in part on a local dataset stored on the client computing device to obtain an update matrix that is descriptive of updated values for the set of parameters of the machine-learned model, wherein the local dataset comprises data descriptive of an input received by the client computing device from a user of the client computing device, wherein training the machine-learned model based at least in part on the local dataset comprises:

training the machine-learned model based at least in part on the local dataset such that updated values are determined only for a pre-selected portion of the set of parameters, and wherein the update matrix is descriptive of only the updated values for the pre-selected portion of the set of parameters; and

communicating information descriptive of the update matrix to a server computing device.

10 . The client computing device of claim 9 , wherein communicating information descriptive of the update matrix to a server computing device comprises subsampling the update matrix to obtain the information descriptive of the update matrix.

11 . The client computing device of claim 10 , wherein subsampling the update matrix comprises:

generating a parameter mask that specifies a portion of the set of parameters to be sampled; and

subsampling the update matrix according to the parameter mask.

12 . The client computing device of claim 11 , wherein generating the parameter mask comprises generating the parameter mask based at least in part on a seed and a pseudo-random number generator, wherein both the client computing device and the server computing device have knowledge of the seed such that the parameter mask is reproducible by the server computing device.

13 . The client computing device of claim 9 , wherein communicatin information descriptive of the update matrix to a server computing device comprises probabilistically quantizing one or more values included in the update matrix.

14 . The client computing device of claim 9 , wherein communicating information descriptive of the update matrix to a server computing device comprises performing probabilistic binary quantization for one or more values included in the update matrix to change each of the one or more values to a maximum value included in the update matrix or a minimum value included in the update matrix.

15 . The client computing device of claim 9 , wherein communicating information descriptive of the update matrix to a server computing device comprises:

defining a plurality of intervals between a maximum value included in the update matrix and a minimum value included in the update matrix; and

probabilistically changing each of one or more values included in the update matrix to a local interval maximum or a local interval maximum.

16 . The client computing device of claim 9 , wherein communicating information descriptive of the update matrix to a server computing device comprises multiplying a vector of the update matrix by a rotation matrix to obtain a rotated update.

17 . The client computing device of claim 16 , wherein communicating information descriptive of the update matrix to a server computing device further comprises:

probabilistically quantizing one or more values included in the rotated update.

18 . The client computing device of claim 17 , wherein the rotation matrix is a structured rotation matrix that does not require complete generation of the rotation matrix by the client computing device.

19 . At least one non-transitory computer-readable medium that stores instructions that, when executed by a client computing device, cause the client computing device to perform operations, the operations comprising:

obtaining global values for a set of parameters of a machine-learned model, wherein the machine-learned model is stored on the client computing device;

training the machine-learned model stored on the client computing device based at least in part on a local dataset stored on the client computing device to obtain an update matrix that is descriptive of updated values for the set of parameters of the machine-learned model, wherein the local dataset comprises data descriptive of an input received by the client computing device from a user of the client computing device, wherein training the machine-learned model based at least in part on the local dataset comprises:

training the machine-learned model based at least in part on the local dataset such that updated values are determined only for a pre-selected portion of the set of parameters, and wherein the update matrix is descriptive of only the updated values for the pre-selected portion of the set of parameters; and

communicating information descriptive of the update matrix to a server computing device.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 12, 2023
From: MCMAHAN, HUGH BRENDAN; BACON, DAVID MORRIS; KONECNY, JAKUB; YU, XINNAN
To: GOOGLE INC.
Reel/Frame 064878/0228 →
CHANGE OF NAME Recorded Sep 12, 2023
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 064880/0823 →
Continuity (4)
Continuation 16850053 · Apr 16, 2020
Continuation 16335695 · Sep 7, 2017
Provisional Application 62400019 · Sep 26, 2016
Related Publication 20230376856A1 · Nov 23, 2023
References Cited (124)
US 6687653B1 · Kurien et al. · 2004 [cited by applicant]
US 6708163B1 · Kargupta et al. · 2004 [cited by applicant]
US 6879944B1 · Tipping et al. · 2005 [cited by applicant]
US 7016529B2 · Simard et al. · 2006 [cited by applicant]
US 7069256B1 · Campos · 2006 [cited by applicant]
US 7664249B2 · Horvitz et al. · 2010 [cited by applicant]
US 8018874B1 · Owechko · 2011 [cited by applicant]
US 8239396B2 · Byun et al. · 2012 [cited by applicant]
US 8321412B2 · Yang et al. · 2012 [cited by applicant]
US 8429103B1 · Aradhye et al. · 2013 [cited by applicant]
US 8954357B2 · Faddoul et al. · 2015 [cited by applicant]
US 9190055B1 · Kiss et al. · 2015 [cited by applicant]
US 9275398B1 · Kumar et al. · 2016 [cited by applicant]
US 9336483B1 · Abeysooriya et al. · 2016 [cited by applicant]
US 9390370B2 · Kingsbury · 2016 [cited by applicant]
US 9400955B2 · Garimella · 2016 [cited by applicant]
US 9424836B2 · Lee et al. · 2016 [cited by applicant]
US 11132602B1 · Xie · 2021 [cited by examiner]
US 11188821B1 · Kalakrishnan · 2021 [cited by examiner]
US 11922313B2 · Rouhani · 2024 [cited by examiner]
US 20030236662A1 · Goodman · 2003 [cited by applicant]
US 20050138571A1 · Keskar et al. · 2005 [cited by applicant]
US 20060224579A1 · Zheng · 2006 [cited by applicant]
US 20080209031A1 · Zhu et al. · 2008 [cited by applicant]
US 20100132044A1 · Kogan et al. · 2010 [cited by applicant]
US 20110085546A1 · Capello et al. · 2011 [cited by applicant]
US 20120016816A1 · Yanase et al. · 2012 [cited by applicant]
US 20120226639A1 · Burdick et al. · 2012 [cited by applicant]
US 20120310870A1 · Caves et al. · 2012 [cited by applicant]
US 20140214735A1 · Harik · 2014 [cited by applicant]
US 20150186798A1 · Vasseur et al. · 2015 [cited by applicant]
US 20150193695A1 · Cruz Mota et al. · 2015 [cited by applicant]
US 20150195144A1 · Vasseur et al. · 2015 [cited by applicant]
US 20150242760A1 · Miao et al. · 2015 [cited by applicant]
US 20150324690A1 · Chilimbi et al. · 2015 [cited by applicant]
US 20180075347A1 · Alistarh et al. · 2018 [cited by applicant]
US 20230376856A1 · McMahan et al. · 2023 [cited by applicant]
CN 1434406 · 2003 [cited by applicant]
CN 103093445 · 2015 [cited by applicant]
CN 105528620 · 2016 [cited by applicant]
WO WO2015126858 · 2015 [cited by applicant]
Kanti et al. “Prediction of bead geometry in pulsed GMA welding using back propagation neural network”, J. MPT, 2008, pp. 300-305. [cited by examiner]
Ailon et al., “Approximate Nearest Neighbors and the Fast Johnson-Lindenstrauss Transform”, 38 [cited by applicant]
Alistarh et al., “QSGB: Randomized Quantization for Communication-Optimal Stochastic Gradient Descent”, arXiv:610.02132v1, , Oct. 7, 2016, 22 pages. [cited by applicant]
Al-Rfou et al., “Conversational Contextual Cues: The Case of Personalization and History for Response Ranking”, arXiv:1606.00372v1, Jun. 1, 2016, 10 pages. [cited by applicant]
Arjevani et al., “Communication Complexity of Distributed Convex Learning and Optimization”, Neural Information Processing Systems, Montreal, Canada, Dec. 7-12, 2015, 9 pages. [cited by applicant]
Balcan et al., “Distributed Learning, Communication Complexity and Privacy”, Conference on Learning Theory, Edinburgh, Scotland, Jun. 25-27, 2012. [cited by applicant]
Bonawitz et al., “Practical Secure Aggregation for Federated Learning on User-Held Data”, arXiv1611.04482v1, Nov. 14, 2016, 5 pages. [cited by applicant]
Braverman et al., “Communication Lower Bounds for Statistical Estimation Problems via a Distributed Data Processing Inequality”, 48 [cited by applicant]
Chaudhuri et al., “Differentially Private Empirical Risk Minimization”, Journal of Machine Learning Research, vol. 12, Jul. 12, 2011, pp. 1069-1109. [cited by applicant]
Chen et al., “Revisiting Distributed Synchronous SGD”, arXiv:1604.00981v3, Mar. 21, 2017, 10 pages. [cited by applicant]
Chen et al., “Communication-Optimal Distributed Clustering”, Neural Information Processing Systems, Barcelona, Spain, Dec. 5-10, 2016, 9 pages. [cited by applicant]
Chilimbi et al., “Project Adam: Building an Efficient and Scalable Deep Learning Training System”, 11th USENIX Symposium on Operating Systems Design and Implementation, Broomfield, Colorado, Oct. 6-8, 2014, pp. 571-582. [cited by applicant]
Courbariaux et al., “BinaryConnect: Training Deep Neural Networks with binary weights during propagations”, NIPS, 2015, pp. 9. [cited by applicant]
Dasgupta et al., “An Elementary Proof of a Theorem of Johnson and Lindenstrauss”, Random Structures & Algorithms, vol. 22, Issue 1, 2003, pp. 60-65. [cited by applicant]
Dean et al., “Large Scale Distributed Deep Networks”, Neural Information Processing Systems, Dec. 3-8, 2012, Lake Tahoe, 9 pages. [cited by applicant]
Denil et al., “Predicting Parameters in Deep Learning”,26th International Conference on Neural Information Processing Systems, Lake Tahoe, Nevada, Dec. 5-10, 2013, pp. 2148-2156. [cited by applicant]
Duchi et al., “Privacy Aware Learning”, arXiv:1210.2085v2, Oct. 10, 2013, 60 pages. [cited by applicant]
Dwork et al., “The Algorithmic Foundations of Differential Privacy”, Foundations and Trends in Theoretical Computer Science, vol. 9, Nos. 3-4, 2014, pp. 211-407. [cited by applicant]
Efron et al., “The Jackknife Estimate of Variance”, The Annals of Statistics, vol. 9, Issue 3, May 1981, pp. 586-596. [cited by applicant]
Elias, “Universal Codeword Sets and Representations of the Integers”, IEEE Transactions on Information Theory, vol. 21, Issue 2, Mar. 1975, pp. 194-203. [cited by applicant]
Extended European Search Report for 19215787.3, mailed on May 6, 2020, 11 pages. [cited by applicant]
Falahatgar et al., “Universal Compression of Power-Law Distributions”, arXiv:1504.08070v2, May 1, 2015, 20 pages. [cited by applicant]
Fercoq et al., “Fast Distributed Coordinate Descent for Non-Strongly Convex Losses”, arXiv:1405.5300v1, May 21, 2014, 6 pages. [cited by applicant]
Gamal et al., “On Randomized Distributed Coordinate Descent with Quantized Updates”, arXiv:1609.05539v1, Sep. 18, 2016, 5 pages. [cited by applicant]
Garg et al., “On Communication Cost of Distributed Statistical Estimation and Dimensionality”, Neural Information Processing Systems, Montreal, Canada, Dec. 8-13, 2014, 9 pages. [cited by applicant]
Golovin et al., “Large-Scale Learning with Less Ram via Randomization”, arXiv:1303.4664v1, Mar. 19, 2013, 10 pages. [cited by applicant]
Han et al., “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding”, arXiv:1510.00149v5, Nov. 20, 2015, 13 pages. [cited by applicant]
Ho et al., “More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server”, Neural Information Processing Systems, Dec. 5-10, 2013, Lake Tahoe, 9 pages. [cited by applicant]
Horadam, “Hadamard Matrices and Their Applications”, Princeton University Press, 2007. [cited by applicant]
International Preliminary Report on Patentability for PCT/US2017/050433, mailed on Sep. 24, 2018, 16 pages. [cited by applicant]
International Search Report and Written Opinion for PCT/US2017/050433, mailed on Nov. 23, 2017, 13 pages. [cited by applicant]
Jaggi et al., “Communication-Efficient Distributed Dual Coordinate Ascent”, arXiv:1409.1458v2, Sep. 29, 2014, 15 pages. [cited by applicant]
Johnson et al., “Accelerating Stochastic Gradient Descent Using Predictive Variance Reduction” Advances in Neural Information Processing Systems, Lake Tahoe, Nevada, Dec. 5-10, 2013, pp. 315-323. [cited by applicant]
Kalantidis et al., “Locally Optimized Product Quantization for Approximate Nearest Neighbor Search”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, 8 pages. [cited by applicant]
Konecny et al., “Federated Optimization: Distributed Machine Learning for On-Device Intelligence”, arXiv:1610.02527v1, Oct. 8, 2016, 38 pages. [cited by applicant]
Konecny et al., “Federated Optimization: Distributed Optimization Beyond the Datacenter”, arXiv:1511.03575v1, Nov. 11, 2015, 5 pages. [cited by applicant]
Konecny et al., “Semi-Stochastic Gradient Descent Methods”, arXiv:1312.1666v1, Dec. 5, 2013, 19 pages. [cited by applicant]
Konecny et al., “Federated Learning: Strategies for Improving Communication Efficiency”, arXiv:610.05492v1, Oct. 18, 2016, 5 pages. [cited by applicant]
Konecny et al., “Randomized Distributed Mean Estimation: Accuracy vs. Communication”, arXiv:1611.07555v1, Nov. 22, 2016, 19 pages. [cited by applicant]
Krichevsky et al., “The Performance of Universal Encoding”, IEEE Transactions on Information Theory, vol. 27, Issue 2, Mar. 1981, pp. 199-207. [cited by applicant]
Krizhevsky, “Learning Multiple Layers of Features from Tiny Images”, Techncal Report, Apr. 8, 2009, 60 pages. [cited by applicant]
Krizhevsky, “One Weird Trick for Parallelizing Convolutional Neural Networks”, arXiv:1404.59997v2, Apr. 26, 2014, 7 pages. [cited by applicant]
Kumar et al., “Fugue: Slow-Worker-Agnostic Distributed Learning for Big Models on Big Data”, Journal of Machine Learning Research: Workshop and Conference Proceedings, Apr. 2014, 9 pages. [cited by applicant]
Li et al., “Ternary Weight Networks”, May 2016, http://arxiv.org/pdf/1605.04711v1, pp. 9. [cited by applicant]
Livni et al., “An Algorithm for Training Polynomial Networks”, arXiv:1304.7045v1, Apr. 26, 2013, 22 pages. [cited by applicant]
Livni et al., “On the Computational Efficiency of Training Neural Networks” arXiv:1410.1141v2, Oct. 28, 2014, 15 pages. [cited by applicant]
Lloyd, “Least Squares Quantization in PCM”, IEEE Transactions on Information Theory, vol. 28, Issue 2, Mar. 1982, pp. 129-137. [cited by applicant]
Ma et al., “Adding vs. Averaging in Distributed Primal-Dual Optimization”, arXiv:1502.03508v2, Jul. 3, 2015, 19 pages. [cited by applicant]
Ma et al., “Distributed Optimization with Arbitrary Local Solvers”, arXiv:1512.04039v2, Aug. 3, 2016, 38 pages. [cited by applicant]
MacKay, “Information Theory, Inference and Learning Algorithms”, Cambridge University Press, 2003. [cited by applicant]
McDonald et al., “Distributed Training Strategies for the Structures Perceptron”, Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, L… [cited by applicant]
McMahan et al., “Communication-Efficient Learning of Deep Networks from Decentralized Data”, arXiv:1602.05629v3, Feb. 28, 2017, 11 pages. [cited by applicant]
McMahan et al., “Federated Learning: Collaborative Machine Learning without Centralized Training Data”, Apr. 6, 2017, https://research.googleblog.com/2017/04/federated-learning-collaborative.html, retrieved on Oct. 3, 2… [cited by applicant]
McMahan et al., “Federated Learning of Deep Networks using Model Averaging”, arXiv:1602.05629, Feb. 17, 2016, 11 pages. [cited by applicant]
Nedic et al., “On Distributed Averaging Algorithms and Quantization Effects”, IEEE Trans. On Autonomatic Control, 2009, pp. 2506-2517. [cited by applicant]
Povey et al., “Parallel Training of Deep Neural Networks with Natural Gradient and Parameter Averaging”, arVix:1410.7455v1, Oct. 27, 2014, 21 pages. [cited by applicant]
Qu et al., “Coordinate Descent with Arbitrary Sampling I: Algorithms and Complexity”, arXiv:1412.8060v2, Jun. 15, 2015, 32 pages. [cited by applicant]
Qu et al., Quartz: Randomized Dual Coordinate Ascent with Arbitrary Sampling, arXiv:1411.5873v1, Nov. 21, 2014, 34 pages. [cited by applicant]
Rabbat et al., “Quantized Incremental Algorithms for Distributed Optimization”, Journal on Selected Areas in Communications, vol. 23, No. 4, 2005, pp. 798-808. [cited by applicant]
Reddi et al., “AIDE: Fast and Communication Efficient Distributed Optimization”, arXiv:1608.06879v1, Aug. 24, 2016, 23 pages. [cited by applicant]
Richtarik et al., “Distributed Coordinate Descent Method for Learning with Big Data”, arXiv:1310.2059v1, Oct. 8, 2013, 11 pages. [cited by applicant]
Seide et al., “1-Bit Stochastic Gradient Descent and Application to Data-Parallel Distributed Training of Speech DNNs”, 15 [cited by applicant]
Shamir et al., “Communication-Efficient Distributed Optimization Using an Approximate Newton-Type Method”, arXiv1312.7853v4, May 13, 2013, 22 pages. [cited by applicant]
Shamir et al., “Distributed Stochastic Optimization and Learning”, 52nd Annual Allerton Conference on Communication, Control, and Computing, Monticello, Illinois, Oct. 1-3, 2014, pp. 850-857. [cited by applicant]
Shokri et al., “Privacy-Preserving Deep Learning” Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, Colorado, Oct. 12-16, 2015, 12 pages. [cited by applicant]
Springenberg et al., “Striving for Simplicity: The All Convolutional Net”, arXiv:1412.6806v3, Apr. 13, 2015, 14 pages. [cited by applicant]
Suresh et al., “Distributed Mean Estimation with Limited Communication”, arXiv.1611.00429v3, Sep. 25, 2017, 17 pages. [cited by applicant]
Tsitsiklis et al., “Communication Complexity of Convex Optimization”, Journal of Complexity, vol. 3, Issue 3, Sep. 1, 1987, pp. 231-243. [cited by applicant]
Wikipedia, “Rounding”, https://en.wikipedia.org/wiki/Rounding, retrieved on Aug. 14, 2017, 13 pages. [cited by applicant]
Woodruff, “Sketching as a Tool for Numerical Linear Algebra”, arXiv:1411.4357v3, Feb. 10, 2015, 139 pages. [cited by applicant]
Xiao-Xuan, “Single-Image Super-Resolution via Low-Rank Matrix Recovery and Joint Learning”, Chinese Journal of Computers, vol. 37, Issue 6, Beijing, Jun. 30, 2014, pp. 1372-1379. [cited by applicant]
Xie et al., “Distributed Machine Learning via Sufficient Factor Broadcasting”, arXiv:1409.5705v2, Sep. 7, 2015, 15 pages. [cited by applicant]
Xing et al., “Petuum: A New Platform for Distributed Machine Learning on Big Data”, Conference on Knowledge Discovery and Data Mining, Aug. 10-13, 2015, Hilton, Sydney, 10 pages. [cited by applicant]
Yadan et al., “Multi-GPU Training of ConvNets”, International Conference on Learning Representations, Apr. 14-16, 2014, Banff, Canada, 4 pages. [cited by applicant]
Yang, “Trading Computation for Communication: Distributed Stochastic Dual Coordinate Ascent”, Advances in Neural Information Processing Systems, Lake Tahoe, Nevada, Dec. 5-10, 2013, pp. 629-637. [cited by applicant]
Yu et al., “Circulant Binary Embedding”, arXiv:1405.3162v1, May 13, 2014, 9 pages. [cited by applicant]
Yu et al., “Orthogonal Random Features”, Neural Information Processing Systems, Barcelona, Spain, Dec. 5-10, 2016, 9 pages. [cited by applicant]
Zhang et al., “Communication-Efficient Algorithms for Statistical Optimization”, arXiv.1209.4129v3, Oct. 11, 2013, 44 pages. [cited by applicant]
Zhang et al., “Communication-Efficient Distributed Optimization of Self Concordant Empirical Loss”, arXiv:1501.00263v1, Jan. 1, 2015, 46 pages. [cited by applicant]
Zhang et al., “DISCO: Distributed Optimization for Self-Concordant Empirical Loss”, 32nd International Conference on Machine Learning, vol. 37, 2015, pp. 362-370. [cited by applicant]
Zhang et al., “Information-Theoretic Lower Bounds for Distributed Statistical Estimation with Communication Constraints”, Neural Information Processing Systems, Lake Tahoe, Nevada, Dec. 5-10, 2013, 9 pages. [cited by applicant]
Zhang et al., “Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines”, arXiv:1512.06216v1, Dec. 19, 2015, 14 pages. [cited by applicant]
Chinese Search Report Corresponding to Application No. 2021109743033 on Jun. 20, 2024. [cited by applicant]