IP Library › Granted Patent US 12,406,185
Granted Patent B1
US 12,406,185 · App. 17/377,188 · Granted Sep 2, 2025

System and method for pruning neural networks at initialization using iteratively conserving synaptic flow

Inventors: Hidenori Tanaka (Sunnyvale, CA); Daniel Kunin (Stanford, CA); Daniel L. K. Yamins (Sunnyvale, CA); Surya Ganguli (Stanford, CA)
Assignees: NTT Research, Inc.; THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY
G06N3/082G06N3/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,406,185
App. No.
17/377,188
Granted
Sep 2, 2025
Kind
B1
Abstract

A system and method to prune parameters of a neural network at initialization uses iterative conserving synaptic flow that saves time, memory and energy both during training and at test time of the neural network. The result of the disclosed pruning system and method are highly sparse trainable subnetworks at initialization, without training and without ever looking at the data (a data agnostic pruning system and method). The pruning system and method preserves the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint.

Claims (36)

1. An apparatus, comprising:

a computer system having an application specific integrated circuit (ASIC) and memory and a plurality of lines of instructions executed by the ASIC to perform operations comprising:

receive an untrained initial neural network having one or more layers with an input layer, an output layer and one or more layers between the input layer and the output layer in which each layer has a plurality of neurons with each neuron having a parameter, a particular neuron in each layer having a synapse that connects to another particular neuron in a different layer of the initial neural network;

initialize a binary mask that prunes one or more neurons in the initial neural network; and

perform iterative synaptic flow pruning to generate a sparse trainable subnetwork with a predetermined level of compression and no layer collapse.

2. The apparatus of claim 1 , wherein the ASIC is further configured to generate a score for each parameter, prune any neuron of the initial neural network whose score does not meet a threshold to generate an updated mask and iteratively generate the score and pruning the one or more neurons until a predetermined number of iterations has been completed.

3. The apparatus of claim 2 , wherein the ASIC is further configured to generate a synaptic saliency score.

4. The apparatus of claim 3 , wherein the ASIC is further configured to generate a positive synaptic saliency score.

5. A method, comprising:

receiving, by an application specific integrated circuit (ASIC), an untrained initial neural network having one or more layers with an input layer, an output layer and one or more layers between the input layer and the output layer in which each layer has a plurality of neurons with each neuron having a parameter, a particular neuron in each layer having a synapse that connects to another particular neuron in a different layer of the initial neural network;

initializing, by the ASIC, a binary mask that prunes one or more neurons in the initial neural network; and

performing, by the ASIC, iterative synaptic flow pruning to generate a sparse trainable subnetwork with a predetermined level of compression and no layer collapse.

6. The method of claim 5 , wherein performing the iterative synaptic flow pruning further comprises:

generating, by the ASIC, a score for each parameter;

pruning, by the ASIC, any neuron of the initial neural network whose score does not meet a threshold to generate an updated mask; and

iteratively generating, by the ASIC, the score and pruning the one or more neurons until a predetermined number of iterations has been completed.

7. The method of claim 6 , wherein the generating the score further comprises generating, by the ASIC, a synaptic saliency score.

8. The method of claim 7 , wherein the generating the score further comprises generating, by the ASIC, a positive synaptic saliency score.

9. An apparatus, comprising:

a computer system having an application specific integrated circuit (ASIC) and memory and a plurality of lines of instructions executed by the ASIC to perform operations comprising:

receive an untrained initial neural network having one or more layers with an input layer, an output layer and one or more layers between the input layer and the output layer in which each layer has a plurality of neurons with each neuron having a parameter, a particular neuron in each layer having a synapse that connects to another particular neuron in a different layer of the initial neural network;

generate a sparse trainable subnetwork with a predetermined level of compression and no layer collapse using iterative synaptic pruning; and

use the sparse trainable subnetwork for data processing.

10. The apparatus of claim 9 , wherein the ASIC is further configured to generate a score for each parameter, prune any neuron of the initial neural network whose score does not meet a threshold to generate an updated mask and iteratively generate the score and pruning the neurons until a predetermined number of iterations has been completed to generate the sparse trainable subnetwork.

11. The apparatus of claim 10 , wherein the ASIC is further configured to generate a synaptic saliency score.

12. The apparatus of claim 11 , wherein the ASIC is further configured to generate a positive synaptic saliency score.

13. A method, comprising:

receiving, by an application specific integrated circuit (ASIC), an untrained initial neural network having one or more layers with an input layer, an output layer and one or more layers between the input layer and the output layer in which each layer has a plurality of neurons with each neuron having a parameter, a particular neuron in each layer having a synapse that connects to another particular neuron in a different layer of the initial neural network;

generating, by the ASIC, a sparse trainable subnetwork with a predetermined level of compression and no layer collapse using iterative synaptic pruning; and

using, by the ASIC, the sparse trainable subnetwork for data processing.

14. The method of claim 13 , wherein generating the sparse trainable subnetwork further comprises:

generating, by the ASIC, a score for each parameter;

pruning, by the ASIC, any neuron of the initial neural network whose score does not meet a threshold to generate an updated mask; and

iteratively generating, by the ASIC, the score and pruning the neurons until a predetermined number of iterations has been completed.

15. The method of claim 14 , wherein the generating the score further comprises generating, by the ASIC, a synaptic saliency score.

16. The method of claim 15 , wherein the generating the score further comprises generating, by the ASIC, a positive synaptic saliency score.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 30, 2025
From: TANAKA, HIDENORI; KUNIN, DANIEL
To: NTT RESEARCH, INC.
Reel/Frame 071267/0843 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 30, 2025
From: YAMINS, DANIEL L.K.; GANGULI, SURYA
To: THE BOARD OF TRUSTEES OF THE LELAND STANFORD JUNIOR UNIVERSITY
Reel/Frame 071267/0930 →
Continuity (1)
Provisional Application 63052317 · Jul 15, 2020
References Cited (300)
US 4941176A · Matyas et al. · 1990 [cited by applicant]
US 6575902B1 · Burton · 2003 [cited by applicant]
US 7610624B1 · Brothers et al. · 2009 [cited by applicant]
US 7703128B2 · Cross et al. · 2010 [cited by applicant]
US 7912698B2 · Statnikov et al. · 2011 [cited by applicant]
US 8135718B1 · Das et al. · 2012 [cited by applicant]
US 8418249B1 · Nuycci et al. · 2013 [cited by applicant]
US 8495747B1 · Nakawatase et al. · 2013 [cited by applicant]
US 8621203B2 · Ekberg et al. · 2013 [cited by applicant]
US 8719924B1 · Williamson et al. · 2014 [cited by applicant]
US 8726379B1 · Stiansen et al. · 2014 [cited by applicant]
US 8762298B1 · Ranjan et al. · 2014 [cited by applicant]
US 8806647B1 · Daswani et al. · 2014 [cited by applicant]
US 8831228B1 · Agrawal et al. · 2014 [cited by applicant]
US 8892876B1 · Huang et al. · 2014 [cited by applicant]
US 9144389B2 · Srinivasan et al. · 2015 [cited by applicant]
US 9183387B1 · Altman et al. · 2015 [cited by applicant]
US 9258321B2 · Amsler et al. · 2016 [cited by applicant]
US 9270689B1 · Wang et al. · 2016 [cited by applicant]
US 9323928B2 · Agarwal et al. · 2016 [cited by applicant]
US 9356942B1 · Joffe · 2016 [cited by applicant]
US 9674880B1 · Egner et al. · 2017 [cited by applicant]
US 9680855B2 · Schultz et al. · 2017 [cited by applicant]
US 9716728B1 · Tumulak · 2017 [cited by applicant]
US 9742778B2 · O'Sullivan et al. · 2017 [cited by applicant]
US 9787640B1 · Xie et al. · 2017 [cited by applicant]
US 10026330B2 · Burford · 2018 [cited by applicant]
US 10038723B2 · Gustafsson · 2018 [cited by applicant]
US 10104097B1 · Yumer et al. · 2018 [cited by applicant]
US 10140381B2 · Trikha et al. · 2018 [cited by applicant]
US 10268820B2 · Okano et al. · 2019 [cited by applicant]
US 10326744B1 · Nossik et al. · 2019 [cited by applicant]
US 10389753B2 · Kawashima et al. · 2019 [cited by applicant]
US 10462159B2 · Inoue et al. · 2019 [cited by applicant]
US 10566084B2 · Kataoka · 2020 [cited by applicant]
US 10644878B2 · Yamamoto · 2020 [cited by applicant]
US 10652270B1 · Hu et al. · 2020 [cited by applicant]
US 10681080B1 · Chen et al. · 2020 [cited by applicant]
US 10887324B2 · Kataoka et al. · 2021 [cited by applicant]
US 20020052858A1 · Goldman et al. · 2002 [cited by applicant]
US 20020138492A1 · Kil · 2002 [cited by applicant]
US 20030163684A1 · Fransdonk · 2003 [cited by applicant]
US 20030163686A1 · Ward et al. · 2003 [cited by applicant]
US 20030169185A1 · Taylor · 2003 [cited by applicant]
US 20030188181A1 · Kunitz et al. · 2003 [cited by applicant]
US 20040015579A1 · Cooper et al. · 2004 [cited by applicant]
US 20040022390A1 · McDonald et al. · 2004 [cited by applicant]
US 20040128535A1 · Cheng · 2004 [cited by applicant]
US 20040158350A1 · Ostergaard et al. · 2004 [cited by applicant]
US 20040267413A1 · Keber · 2004 [cited by applicant]
US 20060037080A1 · Maloof · 2006 [cited by applicant]
US 20060038818A1 · Steele · 2006 [cited by applicant]
US 20060187060A1 · Colby · 2006 [cited by applicant]
US 20060187061A1 · Colby · 2006 [cited by applicant]
US 20060265751A1 · Cosquer et al. · 2006 [cited by applicant]
US 20070136607A1 · Launchbury et al. · 2007 [cited by applicant]
US 20070261112A1 · Todd et al. · 2007 [cited by applicant]
US 20070266433A1 · Moore · 2007 [cited by applicant]
US 20080098479A1 · O'Rourke et al. · 2008 [cited by applicant]
US 20080119958A1 · Bear et al. · 2008 [cited by applicant]
US 20080148398A1 · Mezack et al. · 2008 [cited by applicant]
US 20080220740A1 · Shatzkamer et al. · 2008 [cited by applicant]
US 20080244008A1 · Wilkinson et al. · 2008 [cited by applicant]
US 20080276317A1 · Chandola et al. · 2008 [cited by applicant]
US 20080279387A1 · Gassoway · 2008 [cited by applicant]
US 20080294019A1 · Tran · 2008 [cited by applicant]
US 20080307526A1 · Chung et al. · 2008 [cited by applicant]
US 20080319591A1 · Markiton et al. · 2008 [cited by applicant]
US 20090021394A1 · Coughlin · 2009 [cited by applicant]
US 20090028141A1 · Duong et al. · 2009 [cited by applicant]
US 20090062623A1 · Cohen et al. · 2009 [cited by applicant]
US 20090066521A1 · Atlas et al. · 2009 [cited by applicant]
US 20090067923A1 · Whitford · 2009 [cited by applicant]
US 20090077666A1 · Chen et al. · 2009 [cited by applicant]
US 20090157057A1 · Ferren et al. · 2009 [cited by applicant]
US 20090167531A1 · Ferguson · 2009 [cited by applicant]
US 20090254973A1 · Kwan · 2009 [cited by applicant]
US 20090254992A1 · Schultz et al. · 2009 [cited by applicant]
US 20090287706A1 · Bourges-Waldegg et al. · 2009 [cited by applicant]
US 20100007489A1 · Misra et al. · 2010 [cited by applicant]
US 20100066509A1 · Okuizumi et al. · 2010 [cited by applicant]
US 20100183211A1 · Meetz et al. · 2010 [cited by applicant]
US 20100201489A1 · Griffin · 2010 [cited by applicant]
US 20100246827A1 · Lauter et al. · 2010 [cited by applicant]
US 20100286572A1 · Moersdorf et al. · 2010 [cited by applicant]
US 20110179492A1 · Markopoulou et al. · 2011 [cited by applicant]
US 20110291803A1 · Bajic et al. · 2011 [cited by applicant]
US 20110299420A1 · Waggener et al. · 2011 [cited by applicant]
US 20120005755A1 · Kitazawa et al. · 2012 [cited by applicant]
US 20120024889A1 · Robertson et al. · 2012 [cited by applicant]
US 20120110328A1 · Pate et al. · 2012 [cited by applicant]
US 20120167210A1 · Garcia et al. · 2012 [cited by applicant]
US 20120278889A1 · El-Moussa · 2012 [cited by applicant]
US 20120324568A1 · Wyatt et al. · 2012 [cited by applicant]
US 20130046696A1 · Radhakrishnan · 2013 [cited by applicant]
US 20130046987A1 · Radhakrishnan · 2013 [cited by applicant]
US 20130074186A1 · Muttik · 2013 [cited by applicant]
US 20130104238A1 · Balson et al. · 2013 [cited by applicant]
US 20130111036A1 · Ozawa et al. · 2013 [cited by applicant]
US 20130195326A1 · Bear et al. · 2013 [cited by applicant]
US 20130247205A1 · Schrecker et al. · 2013 [cited by applicant]
US 20130298243A1 · Kumar et al. · 2013 [cited by applicant]
US 20130340084A1 · Schrecker et al. · 2013 [cited by applicant]
US 20130347094A1 · Bettini et al. · 2013 [cited by applicant]
US 20140105573A1 · Hanckmann et al. · 2014 [cited by applicant]
US 20140108474A1 · David et al. · 2014 [cited by applicant]
US 20140115707A1 · Bailey, Jr. · 2014 [cited by applicant]
US 20140122370A1 · Jamal et al. · 2014 [cited by applicant]
US 20140136846A1 · Kitze et al. · 2014 [cited by applicant]
US 20140137257A1 · Martinez et al. · 2014 [cited by applicant]
US 20140153478A1 · Kazmi et al. · 2014 [cited by applicant]
US 20140157405A1 · Joll et al. · 2014 [cited by applicant]
US 20140163640A1 · Edgerton et al. · 2014 [cited by applicant]
US 20140181267A1 · Watdkins et al. · 2014 [cited by applicant]
US 20140181973A1 · Lee et al. · 2014 [cited by applicant]
US 20140189861A1 · Gupta et al. · 2014 [cited by applicant]
US 20140189873A1 · Elder et al. · 2014 [cited by applicant]
US 20140201374A1 · Ashwood-Smith et al. · 2014 [cited by applicant]
US 20140201836A1 · Amsler · 2014 [cited by applicant]
US 20140219096A1 · Rabie et al. · 2014 [cited by applicant]
US 20140222813A1 · Yang et al. · 2014 [cited by applicant]
US 20140229739A1 · Roth et al. · 2014 [cited by applicant]
US 20140237599A1 · Gertner et al. · 2014 [cited by applicant]
US 20140259170A1 · Amsler · 2014 [cited by applicant]
US 20140317261A1 · Shatzkamer et al. · 2014 [cited by applicant]
US 20140317293A1 · Shatzkamer · 2014 [cited by applicant]
US 20140325231A1 · Hook et al. · 2014 [cited by applicant]
US 20140343967A1 · Baker · 2014 [cited by applicant]
US 20150019710A1 · Shaashua et al. · 2015 [cited by applicant]
US 20150033340A1 · Giokas · 2015 [cited by applicant]
US 20150061867A1 · Engelhard et al. · 2015 [cited by applicant]
US 20150074807A1 · Turbin · 2015 [cited by applicant]
US 20150082308A1 · Kiess et al. · 2015 [cited by applicant]
US 20150088791A1 · Lin et al. · 2015 [cited by applicant]
US 20150096024A1 · Haq et al. · 2015 [cited by applicant]
US 20150163242A1 · Laidlaw et al. · 2015 [cited by applicant]
US 20150178469A1 · Park et al. · 2015 [cited by applicant]
US 20150227964A1 · Yan et al. · 2015 [cited by applicant]
US 20150283036A1 · Aggarwal et al. · 2015 [cited by applicant]
US 20150288541A1 · Farango et al. · 2015 [cited by applicant]
US 20150288767A1 · Fargano et al. · 2015 [cited by applicant]
US 20150317169A1 · Sinha et al. · 2015 [cited by applicant]
US 20150326535A1 · Rao et al. · 2015 [cited by applicant]
US 20150326587A1 · Vissamsetty et al. · 2015 [cited by applicant]
US 20150326588A1 · Vissamsetty et al. · 2015 [cited by applicant]
US 20150333979A1 · Schwengler et al. · 2015 [cited by applicant]
US 20150356451A1 · Gupta et al. · 2015 [cited by applicant]
US 20150381423A1 · Xiang · 2015 [cited by applicant]
US 20150381649A1 · Schultz et al. · 2015 [cited by applicant]
US 20160006642A1 · Chang et al. · 2016 [cited by applicant]
US 20160014147A1 · Zoldi et al. · 2016 [cited by applicant]
US 20160050161A1 · Da et al. · 2016 [cited by applicant]
US 20160057234A1 · Parikh et al. · 2016 [cited by applicant]
US 20160065596A1 · Baliga et al. · 2016 [cited by applicant]
US 20160154960A1 · Sharma et al. · 2016 [cited by applicant]
US 20160156644A1 · Wang et al. · 2016 [cited by applicant]
US 20160156656A1 · Boggs et al. · 2016 [cited by applicant]
US 20160182379A1 · Mehra et al. · 2016 [cited by applicant]
US 20160205106A1 · Yacoub et al. · 2016 [cited by applicant]
US 20160218875A1 · Le Saint et al. · 2016 [cited by applicant]
US 20160241660A1 · Nhu · 2016 [cited by applicant]
US 20160248805A1 · Burns et al. · 2016 [cited by applicant]
US 20160301704A1 · Hassanzadeh et al. · 2016 [cited by applicant]
US 20160301709A1 · Hassanzadeh et al. · 2016 [cited by applicant]
US 20160344587A1 · Hoffmann · 2016 [cited by applicant]
US 20160350539A1 · Oberheide et al. · 2016 [cited by applicant]
US 20160350846A1 · Dintenfass et al. · 2016 [cited by applicant]
US 20160352732A1 · Bamasag et al. · 2016 [cited by applicant]
US 20160364553A1 · Smith et al. · 2016 [cited by applicant]
US 20170063893A1 · Franc et al. · 2017 [cited by applicant]
US 20170093915A1 · Ellis et al. · 2017 [cited by applicant]
US 20170149804A1 · Kolbitsch et al. · 2017 [cited by applicant]
US 20170228651A1 · Yamamoto · 2017 [cited by applicant]
US 20170264597A1 · Pizot et al. · 2017 [cited by applicant]
US 20170310485A1 · Robbins et al. · 2017 [cited by applicant]
US 20170318033A1 · Holland et al. · 2017 [cited by applicant]
US 20170366571A1 · Boyer · 2017 [cited by applicant]
US 20170373835A1 · Yamamoto · 2017 [cited by applicant]
US 20170374084A1 · Inoue et al. · 2017 [cited by applicant]
US 20180078213A1 · Kataoka et al. · 2018 [cited by applicant]
US 20180078452A1 · Kataoka et al. · 2018 [cited by applicant]
US 20180083988A1 · Kataoka et al. · 2018 [cited by applicant]
US 20180212768A1 · Kawashima et al. · 2018 [cited by applicant]
US 20180212941A1 · Yamamoto et al. · 2018 [cited by applicant]
US 20180337958A1 · Nagarkar · 2018 [cited by applicant]
US 20190052652A1 · Takahashi et al. · 2019 [cited by applicant]
US 20190075455A1 · Coulier · 2019 [cited by applicant]
US 20190156934A1 · Kataoka · 2019 [cited by applicant]
US 20190228274A1 · Georgiadis · 2019 [cited by examiner]
US 20190370384A1 · Dalek et al. · 2019 [cited by applicant]
US 20200177613A1 · Nilangekar et al. · 2020 [cited by applicant]
CN 104618377A · 2015 [cited by applicant]
JP H11328593A · 1999 [cited by applicant]
JP 2000000232A · 2000 [cited by applicant]
JP 2005217808A · 2005 [cited by applicant]
JP 2008167979A · 2008 [cited by applicant]
JP 2009075695A · 2009 [cited by applicant]
JP 2011060288A · 2011 [cited by applicant]
JP 2011095824A · 2011 [cited by applicant]
JP 2014515538A · 2014 [cited by applicant]
JP 2015012492A · 2015 [cited by applicant]
JP WO2015190446A1 · 2017 [cited by applicant]
JP 2008049602A · 2018 [cited by applicant]
JP 2018148267A · 2018 [cited by applicant]
KR 1020080046779A · 2008 [cited by applicant]
WO WO2008117544A1 · 2008 [cited by applicant]
WO WO2010103800A · 2010 [cited by applicant]
“Pruning neural networks without any data by iteratively conserving synaptic flow” Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yami 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Can… [cited by examiner]
Thornton et al., “Auto-WEKA webpage printed regarding algorithm,” Feb. 17, 2015, 2 pages. [cited by applicant]
Ayat, N.E .; Cheriet, M.; Suen, C.Y.; “Automatic Model Selection for the optimization of SVM Kernels,” Mar. 21, 2005, 35 pages. [cited by applicant]
Brodley, Carla E., “Addressing the Selective Superiority Problem: Automatic Algorithm/Model Class Selection,” (1993) (8 pages). [cited by applicant]
Chapelle, Olivier; Vapnik, Vladimir; Bousquet, Olivier; Mukherjee, Sayan; “Choosing Multiple Parameters for Support Vector Machines,” Machine Learning, 46, 131-159, 2002 © 2002 Kluwer Academic Publishers (29 pages). [cited by applicant]
Lee, Jen-Hao and Lin, Chih-Jen, “Automatic Model Selection for Support Vector Machines,” (2000), 16 pages. [cited by applicant]
Smith, Michael R.; Mitchell, Logan; Giraud-Carrier, Christophe; Martinez, Tony; “Recommending Learning Algorithms and Their Associated Hyperparameters,” Jul. 7, 2014 (2 pages). [cited by applicant]
Thornton, Chris. Thesis: “Auto-WEKA: Combined Selection and Hyperparameter Optimization of Supervised Maching Learning Algorithms,” Submitted to the University of British Columbia, Mar. 2014 (75 pages). [cited by applicant]
Thornton, Chris; Hutter, Frank; Hoos, Holger H.; Leyton-Brown, Kevin. “Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms,” Mar. 2013 (9 pages). [cited by applicant]
Wolinski, Christophe; Kuchcinski, Krzysztof. “Automatic Selection of Application—Specific Reconfigurable Processor Extensions.” Design, Automation & Test in Europe Conference (Date '08), Mar. 2008, Munich, Germany, pp. … [cited by applicant]
Workshop Handout edited by Joaquin Vanschoren, Pavel Brazdil, Carlos Soares and Lars Kotthoff, “Meta-Learning and Algorithm Selection Workshop at ECAI 2014,” MetaSel 2014, Aug. 19, 2014 (66 pages). [cited by applicant]
H. Larochelle et al. “An empirical evaluation of deep architectures on problems with many factors of variation” ACM ICML '07, pp. 473-480, (8 pages). [cited by applicant]
J. Bergstra et al. “Random Search for Hyper-Parameter Optimization” Journal of Machine Learning Research 13 (2012), pp. 281-305 (25 pages). [cited by applicant]
Wikipedia—anonymous, 5 pages. https://en.wikipedia.org/wiki/Decision_tree. [cited by applicant]
Wikipedia—anonymous, 16 pages. en.wikipedia.org/wiki/Support_vector_machine. [cited by applicant]
Wikipedia—anonymous, 11 pages. en.wikipedia.org/wiki/K-nearest_neighbors_algorithm. [cited by applicant]
Wikipedia—anonymous, 8 pages. en.wikipedia.org/wiki/Gradient_boosting. [cited by applicant]
Wikipedia—anonymous, 10 pages. en.wikipedia.org/wiki/Naive_Bayes_classifier. [cited by applicant]
Wikipedia—anonymous, 3 pages. en.wikipedia.org/wiki/Bootstrap_aggregating. [cited by applicant]
Wikipedia—anonymous, 14 pages. en.wikipedia.org/wiki/Logistic_regression. [cited by applicant]
Wikipedia—anonymous, 12 pages. en.wikipedia.org/wiki/AdaBoost. [cited by applicant]
KAGGLE, 2 pages. https://www.kaggle.com/wiki/Home. [cited by applicant]
Wikipedia—anonymous—TLS: Transport Layer Security Protocol, (35 pages)—Webpage https://en.wikipedia.org/wiki/Transport_Layer_security. [cited by applicant]
NIST—National Institute of Standards and Technology, US Department of Commerce “Computer Security Resource Center” AES Algorithm With Galois Counter Mode of Operation, (3 pgs.)—Webpage https:/csrc.nist gov/projects/bloc… [cited by applicant]
Moriarty, et al. PKI Certificate—PKCS #12: Personal Information Exchange Syntax v1.1 (30 pgs.)—Webpage https://tools.ietf.org/html/rfc7292. [cited by applicant]
ITU—International Telecommunication Union—Open Systems Interconnection—X.509: Information Technology—Public-key and attribute framework certificate, (2 pgs.)—Webpage http://www.itu.int/rec/T-REC-X.509/enn. [cited by applicant]
Groves, M., Sakai-Kasahara Key Encryption (SAKKE)—Internet Engineering Task Force dated Feb. 2012, (22 pgs.)—Webpage https://tools.ietf.org/html/rfc6508. [cited by applicant]
Barbosa, L. et al.—SK-KEM: An Identity-Based Kem, Algorithm standardized in IEEE (20 pgs.)—Webpage http://grouper.jeee.org/groups/1363/IBC/submissions/Barbosa-SK-KEM-2006-06.pdf. [cited by applicant]
Boyen-X, et al.—Identity-Based Cryptography Standard (IBCS) #1: Supersingular Curve Implementations of the BF and BB1 Cryptosystems, dated Dec. 2007, (64 pgs.)—Webpage https://tools.ietf.org/html/rfc5091. [cited by applicant]
“Website Traffic, Statistics and Analytics @ Alexa,” an Amazon.com company, 5 pages—Webpage: https://www.alexa.com/siteinfo. [cited by applicant]
Stouffer, K. et al.—“The National Institute of Standards & Technology (NIST) Industrial Control System (ICS) security guide” dated May 2015 (247 pgs.). [cited by applicant]
Chih-Fong et al., Intrusion Detection by Machine Learning: A Review: dated 2009; pp. 11994-12000 (7 pages). [cited by applicant]
Soldo, Fabio, Anh Le, and Athina Markopoulou. “Predictive blacklisting as an implicit recommendation system.” INFOCOM, 2010 Proceedings IEEE. IEEE, 2010. (Year: 2010), 9 pages. [cited by applicant]
Kataoka et al., “Mining Muscle Use Data for Fatigue Reduction in IndyCar”, MIT Sloan Sports Analytics Conference. Mar. 4, 2017 [retrieved Oct. 9, 2018]. Retrieved from the Internet, entire document http://www.sloansport… [cited by applicant]
Kegelman et al., “Insights into vehicle trajectories at the handling limits: analyzing open data from race car drivers; Taylor & Francis, Vehicle System Dynamics” dated Nov. 3, 2016 (18 pgs.). [cited by applicant]
Theodosis et al., “Nonlinear Optimization of a Racing Line for an Autonomous Racecar Using Professional Driving Techniques”, dated Oct. 2012, 7 pages, Citation and abstract, retrieved from the web at: https://www.resear… [cited by applicant]
Tulabandhula et al., “Tire Changes, Fresh Air, and Yellow Flags: Challenges in Predictive Analytics for Professional Racing” MIT, dated Jun. 2014 (16 pages.). [cited by applicant]
Takagahara et al., “hitoe”—A Wearable Sensor Developed through Cross-industrial Collaboration, NTT Technical Review, dated Sep. 4, 2014 (5 pages.). [cited by applicant]
Lee et al., “Development of a Novel Tympanic Temperature Monitoring System for GT Car Racing Athletes,” World Congress on Medical Physics and Biomedical Engineering, May 26-31, 2012, Beijing, China, Abstract Only, pp. 2… [cited by applicant]
NTT Innovation Institute, Inc., Global Cyber Threat Intelligence by Kenji Takahashi, Aug. 16, 2016, retrieved on Aug. 16, 2017, 25 pages, retrieved from the Internet, entire document https://www.slideshare.net/ntti3/glo… [cited by applicant]
How to handle Imbalanced Classification Problems in machine learning? In: Analytics Vidhya. Mar. 17, 2017 (Mar. 17, 2017). Retrieved on Aug. 2, 2019 (Aug. 2, 2019), entire document, 46 pages. https://www.analyticsvidhva… [cited by applicant]
Yen et al., “Cluster-based under-sampling approaches for imbalanced data distributions.” In: Expert Systems with Applications. Apr. 2009 (Apr. 2009. Retrieved on Aug. 2, 2019 (Aug. 2, 2019), entire document), 10 pages. … [cited by applicant]
Chawla et al., “SMOTE: synthetic minority over-sampling technique.” In: Journal of artificial intelligence research. Jun. 2, 2002 (Jun. 2, 2002) Retrieved on Aug. 2, 2019 (Aug. 2, 2019), entire document), 37 pages. http… [cited by applicant]
Malik et al. “Automatic training data cleaning for text classification.” In: 2011 IEEE 11th international conference on data mining workshops. Dec. 11, 2011 (Dec. 11, 2011) Retrieved on Aug. 2, 2019 (Aug. 2, 2019), enti… [cited by applicant]
Moshou et al., “Dynamic muscle fatigue detection using self-organizing maps”, Applied Soft Computing, Elsevier, Amsterdam, NL, vol. 5, No. 4, Jul. 1, 2005 (Jul. 1, 2005), pp. 391-398, 8 pages, XP027669999, ISSN: 1568-49… [cited by applicant]
Zhang et al., “Automated Detection of Driver Fatigue Based on Entropy and Complexity Measures”, IEEE Transactions on Intelligent Transportation Systems, IEEE, Piscataway, NJ, USA, vol. 15, No. 1, Feb. 1, 2014 (Feb. 1, 2… [cited by applicant]
Kiwamori e al., “Getting Started with the Cloud: Machine Learning a comfortable experience with Azure ML”, [Stamped: Received on May 26, 2016, by the Software Information Center] RIC Telecom, 26 pages. [cited by applicant]
Mozer et al., “Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment”, NIPS'88: Proceedings of the 1st International Conference on Neural Information Processing Systems, published Jan… [cited by applicant]
Cun et al., “Optimal Brain Damage”. AT&T Bell Laboratories, Holmdel, N. J. 07733. Abstract. 8 pages. [cited by applicant]
Hassibi et al., “Second order derivatives for network pruning: Optimal Brain Surgeon”, Department of Electrical Engineering, Stanford University, Stanford, CA 94305, 8 pages. [cited by applicant]
Reed et al., “Pruning Algorithms—A Survey”, IEEE Transactions of Neural Networks, vol. 4, No. 5, Sep. 1993, 8 pages. [cited by applicant]
Steven A. Janowsky, “Pruning versus clipping in neural networks”, Department of Physics, Harvard University, Cambridge, Massachusetts 02138 (Received Nov. 30, 1988), 4 pages. [cited by applicant]
Han et al., “Learning both Weights and Connections for Efficient Neural Networks”, Stanford University, 9 pages. [cited by applicant]
Sze et al., “Efficient Processing of Deep Neural Networks a Tutorial and Survey”, IEEE, Aug. 13, 2017, 32 pages. [cited by applicant]
Arora et al., “Stronger Generalization Bounds for Deep Nets via Compression Approach”, Proceedings of the 35th Internationals Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018, 10 pages. [cited by applicant]
Tanaka et al., “From deep learning to mechanistic understanding neuroscience: the structure of retinal prediction”, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, 11 pages. [cited by applicant]
Frankle et al., “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks”, Published as conference paper at ICLR 2019, MIT CSAIL, 42 pages. [cited by applicant]
Frankle et al., “Stabilizing Lottery Ticket Hypothesis”, MIT CSAIL, University of Cambridge Element AI and University of Toronto Vector Institute, Jul. 20, 2020, 13 pages. [cited by applicant]
Morcos et al., “One ticket to win them all: generalizing lottery ticket initialization across datasets and optimizers”, Facebook AI Research, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vanc… [cited by applicant]
Lee et al., “Snip: Single-Shot Network Pruning Based on Connection Sensitivity”, University of Oxford, Published as a conference paper at ICLR 2019, 15 pages. [cited by applicant]
Wang et al., “Picking Winning Tickets Before Training By Preserving Gradient Flow”, Published as a conference paper at ICLR 2020, University of Toronto, Vector Institute, 11 pages. [cited by applicant]
Iandola et al., “Squeezenet: Alexnet-Level Accuracy With 50X Fewer Parameters and <0.5MB Model Size”, Under review as a conference paper at ICLR 2017, DeepScale* & UC Berkeley Stanford University, 13 pages. [cited by applicant]
Howard et al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications”, Google Inc., Apr. 17, 2017, 9 pages. [cited by applicant]
Prabhu et al., “Deep Expander Networks: Efficient Deep Networks from Graph Theory”, Center for Visual Information Technology, Kohli Center on Intelligent Systems, IIIT Hyderabad, India, 16 pages. [cited by applicant]
Jaderberg et al., “Speeding up Convolutional Neural Networks with Low Rank Expansions”, Department of Engineering Science University of Oxford, Oxford, UK, 13 pages. [cited by applicant]
Novikov et al., “Tensorizing Neural Networks”, INRIA, SIERRA project-team, Paris, France; Skolkovo Institute of Science and Technology, National Research University Higher School of Economics, 4Institute of Numerical Ma… [cited by applicant]
Bellec et al., “Deep Rewiring: Training Very Sparse Deep Net-Works”, Published as a conference paper at ICLR 2018, Institute for Theoretical Computer Science, Graz University of Technology, Austria, 24 pages. [cited by applicant]
Mocanu et al., “Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science”, Nature Communications, 2018, 12 pages. [cited by applicant]
Gale et al., “The State of Sparsity in Deep Neural Networks”, This work was completed as part of the Google AI Residency Google Brain and DeepMind, Feb. 25, 2019, 15 pages. [cited by applicant]
Blalock et al., “What is the State of Neural Network Pruning?”, MIT CSAIL, Cambridge, MA, USA. Proceedings of the 3rd MLSys Conference, Austin, TX, USA, 2020, Mar. 6, 2020, 18 pages. [cited by applicant]
Sejun Park*, Jaeho Lee*, Sangwoo Mo, and Jinwoo Shin. Lookahead: A far-sighted alternative of magnitude-based pruning. In International Conference on Learning Representations, 2020, 20 pages. [cited by applicant]
Ehud D Karnin. A simple procedure for pruning back-propagation trained neural networks. IEEE transactions on neural networks, 1(2):239-242, 1990, 4 pages. [cited by applicant]
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016, 17 pages. [cited by applicant]
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11264-1127… [cited by applicant]
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In Advances in neural information processing systems, pp. 1379-1387, 2016, 9 pages. [cited by applicant]
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, pp. 4857-4867, 2017, 11 pages. [cited by applicant]
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. InProceedings of the IEEE Con… [cited by applicant]
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019, 21 pages. [cited by applicant]
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip H. S. Torr. A signal propagation perspective for pruning neural networks at initialization. In International Conference on Learning Representations, 2020, … [cited by applicant]
Wenyuan Zeng and Raquel Urtasun. Mlprune: Multi-layer pruning for automated neural network compression. 2018, 12 pages. [cited by applicant]
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In Proceedings of the 36th International Conference on Machine Learning, vol. 97 of P… [cited by applicant]
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-25… [cited by applicant]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016, 9 pages. [cited by applicant]
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron. In International Conference on Learning Representations, 2019, 15 pages. [cited by applicant]
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS… [cited by applicant]
Seul-Ki Yeom, Philipp Seegerer, Sebastian Lapuschkin, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. Pruning by explaining: A novel criterion for deep neural network pruning. arXiv preprint arXiv:1912.08881, … [cited by applicant]
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems, pp. 3592-3602, 2019, 11 pages. [cited by applicant]
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In Internationa… [cited by applicant]
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. arXiv preprint arXiv:1912.05671, 2019, 30 pages. [cited by applicant]
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. In International Conference on Learning Representations, 2020, 12 pages. [cited by applicant]
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Advances in Neural Information Processing Systems, pp. 384-395, 2018, 12 pages. [cited by applicant]
Stijn Verdenius, Maarten Stol, and Patrick Forré. Pruning via iterative ranking of sensitivity statistics. arXiv preprint arXiv:2006.00896, 2020, 25 pages. [cited by applicant]
Hörandner et al., “Credential: A Framework for Privacy-Preserving Cloud-Based Data Sharing”, 2016 11th International Conference on Availability, Reliability and Security, 8 pages. [cited by applicant]