IP Library › Granted Patent US 12,579,430
Granted Patent B1
US 12,579,430 · App. 17/696,812 · Granted Mar 17, 2026

Decomposition of weight tensors for structural sparsity

Inventors: Eric A. Sather (Palo Alto, CA); Steven L. Teig (Menlo Park, CA)
G06N3/08G06N5/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,579,430
App. No.
17/696,812
Granted
Mar 17, 2026
Kind
B1
Abstract

Some embodiments provide a method for improving structural sparsity of a machine-trained (MT) network. The method receives a network having multiple layers. Each layer of a set of the layers includes multiple filters of weight values. The method replaces the filters of a particular layer of the network with (i) a first set of filters of weight values, (ii) a set of scale values for the first set of filters, and (iii) a second set of filters of weight values. Each scale value corresponds to a different one of the filters of the first set of filters. The method trains the network by applying constraints to bias at least a subset of the scale values towards zero. When a particular scale value falls below a threshold value, the particular scale value is set to zero.

Claims (33)

1 . A method for improving structural sparsity of a machine-trained (MT) network, the method comprising:

receiving a network comprising a plurality of layers, wherein a particular layer of a set of the plurality of layers of the network comprises a plurality of filters of weight values, the plurality of filters comprising a first number of filters, wherein each respective filter of the plurality of filters comprises a first number of weight values;

replacing the plurality of filters of the particular layer of the network with (i) a particular first set of filters of weight values, each filter of the particular first set of filters including the first number of weight values, and wherein the particular first set of filters includes fewer filters than the plurality of filters and (ii) a particular second set of filters of weight values, the particular second set of filters having the first number of filters, each filter of the particular second set of filters including a second number of weight values less than the first number of weight values; and

training the network, wherein training the network comprises:

modifying one or more scale values of a set of scale values each associated with a respective filter of the plurality of filters, and

removing a first filter of the plurality of filters based on comparing a respective scale value of the set of scale values that corresponds to the first filter to a threshold.

2 . The method of claim 1 , wherein training the network comprises, iteratively:

performing stochastic gradient descent (SGD) training to modify weight values of the plurality of filters; and

projecting each scale value of the set of scale values to a set of states, wherein projecting a particular scale value comprises:

projecting the particular scale value to one of (i) a first state in which the particular scale value is set to zero and (ii) a second state in which the particular scale value is not set to zero.

3 . The method of claim 2 , wherein performing the SGD training for a particular iteration comprises:

propagating a set of inputs through the network using weight values and scale values associated with the plurality of filters from a previous projection to generate a set of outputs; and

modifying the weight values and the scale values using a loss function that comprises (i) a term that measures differences between the set of outputs and an expected set of outputs for the set of inputs and (ii) a set of terms that bias at least a subset of the scale values towards zero.

4 . The method of claim 3 , wherein the set of terms comprise Lagrange multipliers for each of the scale values.

5 . The method of claim 2 , wherein the projection is subject to a set of constraints that requires that at least a particular number of the set of scale values be set to zero.

6 . The method of claim 1 further comprising replacing each filter of the plurality of filters of the network with (i) a respective first set of filters of weight values, (ii) a respective set of scale values corresponding to the respective first set of filters, and (iii) a respective second set of filters of weight values.

7 . The method of claim 6 , wherein a constraint is applied when training the network that requires a particular number of scale values to be set to zero such that a threshold number of weight values are removed from the network.

8 . The method of claim 6 , wherein a constraint is applied when training the network that requires that a particular number of scale values are set to zero such that the network is executable in a particular amount of time by a particular neural network inference circuit.

9 . A non-transitory machine-readable medium storing a program which when executed by at least one processing unit improves structural sparsity of a machine-trained (MT) network, the program comprising sets of instructions for:

receiving a network comprising a plurality of layers, wherein a particular layer of a set of the plurality of layers of the network comprises a plurality of filters, the plurality of filters comprising a first number of filters, wherein each respective filter of the plurality of filters comprises a first number of weight values;

replacing the plurality of filters of the particular layer of the network with (i) a particular first set of filters of weight values, each filter of the particular first set of filters including the first number of weight values, and wherein the particular first set of filters includes fewer filters than the plurality of filters and (ii) a particular second set of filters of weight values, the particular second set of filters having the first number of filters, each filter of the particular second set of filters including a second number of weight values less than the first number of weight values; and

training the network, wherein training the network comprises:

modifying one or more scale values of a set of scale values each associated with a respective filter of the plurality of filters, and

removing a first filter of the plurality of filters based on comparing a respective scale value of the set of scale values that corresponds to the first filter to a threshold.

10 . The non-transitory machine-readable medium of claim 9 , wherein the set of instructions for training the network comprises sets of instructions for, iteratively:

performing stochastic gradient descent (SGD) training to modify weight values of the plurality of filters; and

projecting each scale value of the set of scale values to a set of states, wherein projecting a particular scale value comprises:

projecting the particular scale value to one of (i) a first state in which the particular scale value is set to zero and (ii) a second state in which the particular scale value is not set to zero.

11 . The non-transitory machine-readable medium of claim 10 , wherein the set of instructions for performing the SGD training for a particular iteration comprises sets of instructions for:

propagating a set of inputs through the network using weight values and scale values associated with the plurality of filters from a previous projection to generate a set of outputs; and

modifying the weight values and the scale values using a loss function that comprises (i) a term that measures differences between the set of outputs and an expected set of outputs for the set of inputs and (ii) a set of terms that bias at least a subset of the scale values towards zero.

12 . The non-transitory machine-readable medium of claim 10 , wherein the projection is subject to a set of constraints that requires that at least a particular number of the set of scale values be set to zero.

13 . The non-transitory machine-readable medium of claim 9 , wherein the program further comprises a set of instructions for replacing-each filter of the plurality of filters of the network with (i) a respective first set of filters of weight values, (ii) a respective set of scale values corresponding to the respective first set of filters, and (iii) a respective second set of filters of weight values.

Assignments (3)
BILL OF SALE Recorded Oct 31, 2024
From: AMAZON.COM SERVICES LLC
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 069288/0490 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 31, 2024
From: PERCEIVE CORPORATION
To: AMAZON.COM SERVICES LLC
Reel/Frame 069288/0731 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 25, 2022
From: SATHER, ERIC A.; TEIG, STEVEN L.
To: PERCEIVE CORPORATION
Reel/Frame 059696/0127 →
Continuity (4)
Continuation In Part 17089653 · Nov 4, 2020
Provisional Application 63189516 · May 17, 2021
Provisional Application 63178889 · Apr 23, 2021
Provisional Application 63065472 · Aug 13, 2020
References Cited (101)
US 9904874B2 · Shoaib et al. · 2018 [cited by applicant]
US 20130138589A1 · Yu et al. · 2013 [cited by applicant]
US 20160086078A1 · Ji et al. · 2016 [cited by applicant]
US 20160174902A1 · Georgescu et al. · 2016 [cited by applicant]
US 20170286830A1 · El-Yaniv et al. · 2017 [cited by applicant]
US 20180107925A1 · Choi et al. · 2018 [cited by applicant]
US 20180197049A1 · Tran et al. · 2018 [cited by applicant]
US 20190012594A1 · Fukuda · 2019 [cited by examiner]
US 20190042948A1 · Lee et al. · 2019 [cited by applicant]
US 20190065896A1 · Lee · 2019 [cited by examiner]
US 20190138882A1 · Choi et al. · 2019 [cited by applicant]
US 20190138896A1 · Deng · 2019 [cited by applicant]
US 20190147323A1 · Li · 2019 [cited by examiner]
US 20190171927A1 · Diril et al. · 2019 [cited by applicant]
US 20190180184A1 · Deng · 2019 [cited by examiner]
US 20190188557A1 · Lowell et al. · 2019 [cited by applicant]
US 20190228274A1 · Georgiadis et al. · 2019 [cited by applicant]
US 20190286970A1 · Karaletsos · 2019 [cited by examiner]
US 20190340492A1 · Burger et al. · 2019 [cited by applicant]
US 20190354842A1 · Louizos et al. · 2019 [cited by applicant]
US 20190362235A1 · Xu · 2019 [cited by applicant]
US 20200005143A1 · Zamora Esquivel · 2020 [cited by examiner]
US 20200104692A1 · Hill · 2020 [cited by examiner]
US 20200134461A1 · Chai et al. · 2020 [cited by applicant]
US 20200202213A1 · Rouhani et al. · 2020 [cited by applicant]
US 20200202218A1 · Csefalvay · 2020 [cited by applicant]
US 20200210838A1 · Lo et al. · 2020 [cited by applicant]
US 20200264876A1 · Lo et al. · 2020 [cited by applicant]
US 20200302269A1 · Ovtcharov et al. · 2020 [cited by applicant]
US 20210019630A1 · Yao · 2021 [cited by examiner]
US 20210042626A1 · Krishnamoorthy · 2021 [cited by examiner]
US 20210073644A1 · Lin · 2021 [cited by examiner]
US 20210224642A1 · Moriya · 2021 [cited by examiner]
US 20210248459A1 · Li · 2021 [cited by examiner]
US 20210264271A1 · Gebre · 2021 [cited by examiner]
US 20210406672A1 · Hoang et al. · 2021 [cited by applicant]
Wang, “Fixed-point Factorized Networks”, 2017 IEEE Conference on Computer Vision and Pattern Recognition. (Previously supplied). (Year: 2017). [cited by examiner]
Sze, “Efficient Processing of Deep Neural Networks: A Tutorial and Survey”, IEEE, 2017. (Previously supplied). (Year: 2017). [cited by examiner]
Achterhold, Jan, et al., “Variational Network Quantization,” Proceedings of 6th International Conference on Learning Representations (ICLR 2018), Apr. 30-May 3, 2018, 18 pages, ICLR, Vancouver, BC, Canada. [cited by applicant]
Andri, Renzo, et al., “YodaNN: An Architecture for Ultra-Low Power Binary-Weight CNN Acceleration,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Mar. 14, 2017, 14 pages, IEEE, New York,… [cited by applicant]
Bagherinezhad, Hessam, et al., “LCNN: Look-up Based Convolutional Neural Network,” Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Jul. 21-26, 2017, 10 pages, IEEE, Honolulu, … [cited by applicant]
Bong, Kyeongryeol, et al., “A 0.62mW Ultra-Low-Power Convolutional-Neural-Network Face-Recognition Processor and a CIS Integrated with Always-On Haar-Like Face Detector,” Proceedings of 2017 IEEE International Solid-Sta… [cited by applicant]
Boo, Yoonho, et al., “Structured Sparse Ternary Weight Coding of Deep Neural Networks for Efficient Hardware Implementations,” 2017 IEEE Workshop on Signal Processing Systems (SiPS), Oct. 3-5, 2017, 6 pages, IEEE, Lorie… [cited by applicant]
Chen, Yu-Hsin, et al., “Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks,” Proceedings of 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA 2… [cited by applicant]
Chen, Yu-Hsin, et al., “Using Dataflow to Optimize Energy Efficiency of Deep Neural Network Accelerators,” IEEE Micro, Jun. 14, 2017, 10 pages, vol. 37, Issue 3, IEEE, New York, NY, USA. [cited by applicant]
Courbariaux, Matthieu, et al., “Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1,” Mar. 17, 2016, 11 pages, arXiv:1602.02830v3, Computing Research Repository (CoRR… [cited by applicant]
Courbariaux, Matthieu, et al., “BinaryConnect: Training Deep Neural Networks with Binary Weights during Propagations,” Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS 15),… [cited by applicant]
Emer, Joel, et al., “Hardware Architectures for Deep Neural Networks,” CICS/MTL Tutorial, Mar. 27, 2017, 258 pages, Massachusetts Institute of Technology, Cambridge, MA, USA, retrieved from http://www.rle.mit.edu/eems/w… [cited by applicant]
Guo, Yiwen, et al., “Network Sketching: Exploring Binary Structure in Deep CNNs,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Jul. 21-26, 2017, 9 pages, IEEE, Honolulu, HI. [cited by applicant]
He, Zhezhi, et al., “Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy,” Jul. 20, 2018, 8 pages, arXiv:1807.07948v1, Computing Research Repository (CoRR)—Cornell University, Ithaca, NY… [cited by applicant]
Hegde, Kartik, et al., “UCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetition,” Proceedings of the 45th Annual International Symposium on Computer Architecture (ISCA '18), Jun. 2-6, 2018, 14… [cited by applicant]
Huan, Yuxiang, et al., “A Low-Power Accelerator for Deep Neural Networks with Enlarged Near-Zero Sparsity,” May 22, 2017, 5 pages, arXiv:1705.08009v1, Computer Research Repository (CoRR)—Cornell University, Ithaca, NY, … [cited by applicant]
Jain, Anil K., et al., “Artificial Neural Networks: A Tutorial,” Computer, Mar. 1996, 14 pages, vol. 29, Issue 3, IEEE. [cited by applicant]
Jouppi, Norman, P., et al., “In-Datacenter Performance Analysis of a Tensor Processing Unit,” Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA '17), Jun. 24-28, 2017, 17 pages, ACM, … [cited by applicant]
Judd, Patrick, et al., “Cnvlutin2: Ineffectual-Activation-and-Weight-Free Deep Neural Network Computing,” Apr. 29, 2017, 6 pages, arXiv:1705.00125v1, Computer Research Repository (CoRR)—Cornell University, Ithaca, NY, U… [cited by applicant]
Kingma, Diederik P., et al., “Auto-Encoding Variational Bayes,” May 1, 2014, 14 pages, arXiv:1312.6114v10, Computing Research Repository (CoRR)—Cornell University, Ithaca, NY, USA. [cited by applicant]
Kong, Chen, et al., “Take it in your stride: Do we need striding in CNNs?,” Dec. 7, 2017, 9 pages, arXiv:1712.02502v1, Computer Research Repository (CoRR)—Cornell University, Ithaca, NY, USA. [cited by applicant]
Leng, Cong, et al., “Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM,” Proceedings of 32nd AAAI Conference on Artificial Intelligence (AAAI-18), Feb. 2-7, 2018, 16 pages, Association for the Advance… [cited by applicant]
Li, Fengfu, et al., “Ternary Weight Networks,” May 16, 2016, 9 pages, arXiv:1605.04711v1, Computing Research Repository (CoRR)—Cornell University, Ithaca, NY, USA. [cited by applicant]
Merolla, Paul, et al., “Deep Neural Networks are Robust to Weight Binarization and Other Non-linear Distortions,” Jun. 7, 2016, 10 pages, arXiv:1606.01981v1, Computing Research Repository (CoRR)—Cornell University, Itha… [cited by applicant]
Moshovos, Andreas, et al., “Exploiting Typical Values to Accelerate Deep Learning,” Computer, May 24, 2018, 13 pages, vol. 51—Issue 5, IEEE Computer Society, Washington, D.C. [cited by applicant]
Non-Published Commonly Owned U.S. Appl. No. 17/089,648, filed Nov. 4, 2020, 118 pages, Perceive Corporation. [cited by applicant]
Non-Published Commonly Owned U.S. Appl. No. 17/089,653, filed Nov. 4, 2020, 118 pages, Perceive Corporation. [cited by applicant]
Non-Published Commonly Owned U.S. Appl. No. 17/089,660, filed Nov. 4, 2020, 118 pages, Perceive Corporation. [cited by applicant]
Non-Published Commonly Owned Related U.S. Appl. No. 17/696,809 with similar specification, filed Mar. 16, 2022, 109 pages, Perceive Corporation. [cited by applicant]
Non-Published Commonly Owned Related U.S. Appl. No. 17/696,810 with similar specification, filed Mar. 16, 2022, 110 pages, Perceive Corporation. [cited by applicant]
Non-Published Commonly Owned Related U.S. Appl. No. 17/696,819 with similar specification, filed Mar. 16, 2022, 110 pages, Perceive Corporation. [cited by applicant]
Park, Jongsoo, et al., “Faster CNNs with Direct Sparse Convolutions and Guided Pruning,” Jul. 28, 2017, 12 pages, arXiv:1608.01409v5, Computer Research Repository (CoRR)—Cornell University, Ithaca, NY, USA. [cited by applicant]
Rastegari, Mohammad, et al., “XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks,” Proceedings of 2016 European Conference on Computer Vision (ECCV '16), Oct. 8-16, 2016, 17 pages, Lecture Note… [cited by applicant]
Shayer, Oran, et al., “Learning Discrete Weights Using the Local Reparameterization Trick,” Proceedings of 6th International Conference on Learning Representations (ICLR 2018), Apr. 30-May 3, 2018, 12 pages, ICLR, Vanco… [cited by applicant]
Sze, Vivienne, et al., “Efficient Processing of Deep Neural Networks: A Tutorial and Survey,” Aug. 13, 2017, 32 pages, arXiv:1703.09039v2, Computer Research Repository (CoRR)—Cornell University, Ithaca, NY, USA. [cited by applicant]
Vaswani, Sharan, “Exploiting Sparsity in Supervised Learning,” Month Unknown 2014, 9 pages, retrieved from https://vaswanis.github.io > optimization_report. [cited by applicant]
Wang, Min, et al., “Factorized Convolutional Neural Networks,” 2017 IEEE International Conference on Computer Vision Workshops (ICCVW '17), Oct. 22-29, 2017, 9 pages, IEEE, Venice, Italy. [cited by applicant]
Wen, Wei, et al., “Learning Structured Sparsity in Deep Neural Networks,” Oct. 18, 2016, 10 pages, arXiv:1608.03665v4, Computer Research Repository (CoRR)—Cornell University, Ithaca, NY, USA. [cited by applicant]
Yan, Shi, “L1 Norm Regularization and Sparsity Explained for Dummies,” Aug. 27, 2016, 13 pages, retrieved from https://blog.mlreview.com/I1-norm-regularization-and-sparsity-explained-for-dummies-5b0e4be3938a. [cited by applicant]
Yang, Tien-Ju, et al., “Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning,” Apr. 18, 2017, 9 pages, arXiv:1611.05128v4, Computer Research Repository (CoRR)—Cornell University, Ithaca, N… [cited by applicant]
Yang, Xuan, et al., “DNN Dataflow Choice Is Overrated,” Sep. 10, 2018, 13 pages, arXiv:1809.04070v1, Computer Research Repository (CoRR)—Cornell University, Ithaca, NY, USA. [cited by applicant]
Zhang, Dongqing, et al., “LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks,” Jul. 26, 2018, 21 pages, arXiv:1807.10029v1, Computer Research Repository (CoRR)—Cornell University, Ithaca,… [cited by applicant]
Zhou, Shuchang, et al., “DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients,” Jul. 17, 2016, 14 pages, arXiv:1606.06160v2, Computer Research Repository (CoRR)—Cornell University,… [cited by applicant]
Zhu, Chenzhuo, et al., “Trained Ternary Quantization,” Dec. 4, 2016, 9 pages, arXiv:1612.01064v1, Computing Research Repository (CoRR)—Cornell University, Ithaca, NY, USA. [cited by applicant]
Jin, Canran,, et al., “Sparse Ternary Connect: Convolutional Neural Networks Using Ternarized Weights with Enhanced Sparsity,” 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC), Jan. 22-25, 2018, 6… [cited by applicant]
Li, Yawei, et al., “Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression,” Mar. 20, 2020, 14 pages, arXiv:2003.08935, Computer Research Repository (CoRR)—Cornell University, Ithaca,… [cited by applicant]
Wang, Peisong, et al., “Fixed-Point Factorized Networks”, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 21-26, 2017, 9 pages, IEEE, Honolulu, Hawaii, USA. [cited by applicant]
Chen, Shangyu, et al., “Deep Neural Network Quantization via Layer-Wise Optimization Using Limited Training Data,” Proceeding of the 33rd AAAI Conference on Artificial Intelligence (AAAI-19), Jul. 2019, 8 pages, AAAI. [cited by applicant]
He, Yang, et al., “Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks,” Aug. 21, 2018, 8 pages, retrieved from https://arxiv.org/abs/1808.06866. [cited by applicant]
Hu, Yiming, et al., “A Novel Channel Pruning Method for Deep Neural Network Compression,” May 29, 2018, 10 pages, retrieved from https://arxiv.org/abs/1805.11394. [cited by applicant]
Jaderberg, Max, et al., “Speeding Up Convolutional Neural Networks with Low Rank Expansions,” May 15, 2014, 12 pages, retrieved from https://arxiv.org/abs/1405.3866. [cited by applicant]
Kozyrskiy, Nikolay, et al., “CNN Acceleration by Low-Rank Approximation with Quantized Factors,” Jun. 16, 2020, 14 pages, retrieved from https://arxiv.org/abs/2006.08878. [cited by applicant]
Molchanov, Pavlo, et al., “Pruning Convention Neural Networks for Resource Efficient Inference,” Jun. 8, 2017, 17 pages, retrieved from https://arxiv.org/abs/1611.06440. [cited by applicant]
Yang, Huanrui, et al., “Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification,” Apr. 20, 2020, 14 pages, retrieved from https://arxiv.org/abs/2004.0903… [cited by applicant]
Zhang, Xiangyu, et al., “Accelerating Very Deep Convolutional Networks for Classification and Detection,” Nov. 18, 2015, 14 pages, retrieved from https://arxiv.org/abs/1505.06798. [cited by applicant]
Bhattacharya, Sourav, et al., “Sparsification and Separation of Deep Learning Layers for Constrained Resource Inference on Wearables,” SenSys '16, Nov. 14-16, 2016, 15 pages, ACM, Stanford, CA, USA. [cited by applicant]
Han, Song, et al., “Learning Both Weights and Connections for Efficient Neural Networks,” Oct. 30, 2015, 9 pages, retrieved from https://arxiv.org/abs/1506.02626. [cited by applicant]
Hu, Hengyuan, et al., “Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures,” Jul. 12, 2016, 9 pages, retrieved from https://arxiv.org/abs/1607.03250v1. [cited by applicant]
Neklyudov, Kirill, et al., “Structured Bayesian Pruning via Log-Normal Multiplicative Noise,” Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Dec. 4-9, 2017, 10 pages, ACM, Long … [cited by applicant]
Wang, Peiqi, et al., “SNrram: An Efficient Sparse Neural Network Computation Architecture Based on Resistive Random-Access Memory,” DAC '18, Jun. 24-29, 2018, 7 pages, ACM, San Francisco, CA, USA. [cited by applicant]
Agostinelli, Forest, et al., “Learning Activation Functions to Improve Deep Neural Networks,” Apr. 21, 2015, 9 pages, retrieved from https://arxiv.org/abs/1412.6830. [cited by applicant]
Aizenberg, Igor, “Periodic Activation Function and a Modified Learning Algorithm for the Multivalued Neuron,” IEEE Transactions on Neural Networks, Dec. 2010, 11 pages, vol. 21, No. 12, IEEE. [cited by applicant]
Martens, James, “New Insights and Perspectives on the Natural Gradient Method,” Nov. 21, 2017, 59 pages, retrieved from https://arxiv.org/abs/1412.1193v9. [cited by applicant]
Withagen, Heini, “Reducing the Effect of Quantization by Weight Scaling,” Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN '94), Jun. 28-Jul. 2, 1994, 3 pages, IEEE, Orlando, Florida, USA. [cited by applicant]
Zhang, “A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers”, 2018. (Year: 2018). [cited by applicant]