IP Library › Granted Patent US 12,198,040
Granted Patent B2
US 12,198,040 · App. 18/313,278 · Granted Jan 14, 2025

Method for compressing neural network model and electronic apparatus for performing the same

Inventor: Kyunghwan Shim (Seoul, KR)
Assignee: NOTA, INC.
G06N3/063G06V10/764
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,198,040
App. No.
18/313,278
Granted
Jan 14, 2025
Kind
B2
Abstract

A method for compressing a neural network model is disclosed. The method for compressing a neural network model includes receiving, at a processor of the electronic apparatus, an original model including a plurality of layers each including a plurality of filters, a compression ratio to be applied to the original model, and a metric for determining an importance of the plurality of filters, determining the importance of the plurality of filters using the metric, normalizing the importance of the plurality of filters layer by layer, and compressing the original model by removing at least one filter among the plurality of filters based on the normalized importance and the compression ratio.

Claims (66)

1. A method for compressing a neural network model that is performed by an electronic apparatus, comprising:

receiving, an original model including a plurality of layers, a compression ratio to be applied to the original model, and a metric for determining an importance score for a filter, wherein the plurality of layers includes a first layer including a plurality of filters;

determining importance scores for the plurality of filters using the metric;

normalizing the importance scores for the plurality of filters; and

compressing the original model by removing at least one filter among the plurality of filters based on the normalized importance scores and the compression ratio,

wherein the plurality of filters includes:

a first filter having a first importance score,

a plurality of second filters, each having a second importance score greater than or equal to the first importance score, and

at least one third filter having a third importance score smaller than the first importance score, and

wherein the first importance score is normalized based on a sum of the first importance score and the second importance scores of all second filters, excluding the third importance score from the normalization of the first importance score.

2. The method of claim 1 , wherein the determining the importance scores for the plurality of filters includes:

calculating the importance scores for the plurality of filters using the metric, and

adjusting the importance scores for the plurality of filters based on a redundancy of the plurality of filters.

3. The method of claim 2 , wherein a redundancy of the first filter is calculated based on the first importance score and an average and a standard deviation of importance scores for all filters included in the first layer.

4. The method of claim 1 , wherein the original model includes a second layer connected to the first layer via a predetermined operator, and

wherein the at least one filter to be removed from the first layer is determined by additionally using importance scores of a plurality of filters included in the second layer.

5. The method of claim 1 , wherein the removed at least one filter is determined by sorting the plurality of filters in order of the importance scores for the plurality of filters, and selecting a predetermined number of filters in increasing order of an importance score from the plurality of filters, and

wherein the predetermined number is determined based on the compression ratio.

6. The method of claim 4 , wherein the at least one filter is determined based on importance scores for filters with a same index in both the first layer and the second layer, and a policy selected by a user.

7. An electronic apparatus for compressing a neural network model, comprising:

a memory configured to store at least one instruction; and

a processor,

wherein execution of the at least one instruction causes the processor to:

receive an original model including a plurality of layers, a compression ratio to be applied to the original model, and a metric for determining an importance score for a filter, wherein the plurality of layers includes a first layer including a plurality of filters,

determine importance scores for the plurality of filters using the metric,

normalize the importance scores for the plurality of filters layer by layer, and

compress the original model by removing at least one filter among the plurality of filters based on the normalized importance scores and the compression ratio,

wherein the plurality of filters includes:

a first filter having a first importance score,

a plurality of second filters, each having a second importance score greater than or equal to the first importance score, and

at least one third filter having a third importance score smaller than the first importance score, and

wherein the first importance score is normalized based on a sum of the first importance score and the second importance scores of all second filters, excluding the third importance score from the normalization of the first importance score.

8. The electronic apparatus of claim 7 , wherein the processor is further configured to:

calculate the importance scores for the plurality of filters using the metric, and

adjust the importance scores for the plurality of filters based on a redundancy of the plurality of filters.

9. The electronic apparatus of claim 8 , wherein the processor is further configured to calculate a redundancy of the first filter based on the first importance score and an average and a standard deviation of importance scores for all filters included in the first layer.

10. The electronic apparatus of claim 7 , wherein the original model includes a second layer connected to the first layer via a predetermined operator, and

wherein the at least one filter to be removed from the first layer is determined by additionally using importance scores of a plurality of filters included in the second layer.

11. The electronic apparatus of claim 7 , wherein the processor is further configured to:

sort the plurality of filters in order of the importance scores for the plurality of filters, and

select a predetermined number of filters in increasing order of an importance score from the plurality of filters,

wherein the predetermined number is determined based on the compression ratio.

12. The electronic apparatus of claim 10 , wherein the at least one filter is determined based on importance scores for filters with a same index in both the first layer and the second layer, and a policy selected by a user.

13. A method for compressing a neural network model that is performed by an electronic apparatus, comprising:

receiving, at a processor of the electronic apparatus, an original model including a plurality of layers, and a metric for determining an importance score for a filter, wherein the plurality of layers includes a first layer including a plurality of filters;

determining importance scores for the plurality of filters using the metric;

normalizing the importance scores for the plurality of filters layer by layer; and

compressing the original model by removing at least one filter among the plurality of filters in interrelated layers based on the normalized importance scores and a policy, the policy related to a method for removing filters included in the interrelated layers among the plurality of layers,

wherein the plurality of filters includes:

a first filter having a first importance score,

a plurality of second filters having a second importance score greater than or equal to the first importance score, and

at least one third filter having a third importance score smaller than the first importance score, and

wherein the first importance score is normalized based on a sum of the first importance score and the second importance scores of all second filters, excluding the third importance score from the normalization of the first importance score.

14. The method of claim 13 , wherein the interrelated layers include the first layer and a second layer including a plurality of filters, and

wherein the removed at least one filter is determined by:

determining a plurality of first candidate filters in the first layer and a plurality of second candidate filters in the second layer based on the normalized importance scores and a compression ratio configured by a user; and

selecting at least one first target filter in the first layer and at least one second target filter in the second layer based on the policy, the plurality of first candidate filters, and the plurality of second candidate filters.

15. The method of claim 14 , wherein when the policy is a first policy, among the plurality of first candidate filters, a first candidate with an index identical to one of indices of the plurality of second candidate filters is selected as the at least one first target filter, and among the plurality of second candidate filters, a second candidate with an index identical to one of indices of the plurality of first candidate filters is selected as the at least one second target filter.

16. The method of claim 14 , wherein when the policy is a second policy, the at least one first target filter includes a filter with an index identical to one of indices of the plurality of second candidate filters among the plurality of first filters and the plurality of first candidate filters, and the at least one second target filter includes a filter with an index identical to one of indices of the plurality of first candidate filters among the plurality of second filters and the plurality of second candidate filters.

17. The method of claim 14 , wherein when the policy is a third policy, the removed at least one filter is determined based on a sum of importance scores for filters with an identical index in both the first layer and the second layer.

18. The method of claim 14 , wherein when the policy is a fourth policy, the removed at least one filter is determined based on an average of importance scores for filters with an identical index in both the first layer and the second layer.

19. The method of claim 13 , wherein the interrelated layers are connected through a predetermined operator, and the predetermined operator includes an element-wise operator.

20. The method of claim 13 , wherein the determining the importance scores for the plurality of filters includes:

calculating the importance scores for the plurality of filters using the metric, and

adjusting the importance scores for the plurality of filters based on a redundancy of the calculated importance scores for the plurality of filters,

wherein a redundancy of the first filter is calculated based on the first importance score and an average and a standard deviation of importance scores for all filters included in the first layer.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 5, 2023
From: SHIM, KYUNGHWAN
To: NOTA, INC.
Reel/Frame 063557/0024 →
Priority Claims (2)
KR 10-2022-0057599 · May 11, 2022 · national
KR 10-2022-0182086 · Dec 22, 2022 · national
Continuity (1)
Related Publication 20230368010A1 · Nov 16, 2023
References Cited (71)
US 20190286989A1 · Wang et al. · 2019 [cited by applicant]
US 20200193266A1 · Scheidegger et al. · 2020 [cited by applicant]
US 20210073641A1 · Toshiba · 2021 [cited by applicant]
US 20210326710A1 · Wang et al. · 2021 [cited by applicant]
US 20210334663A1 · Shen et al. · 2021 [cited by applicant]
US 20210397963A1 · Jiang et al. · 2021 [cited by applicant]
US 20220237454A1 · Jain et al. · 2022 [cited by applicant]
US 20220253708A1 · Tamada · 2022 [cited by examiner]
CN 110062233A · 2019 [cited by applicant]
CN 110263628A · 2019 [cited by applicant]
CN 111723915A · 2020 [cited by applicant]
CN 111723915B · 2021 [cited by applicant]
CN 113283524A · 2021 [cited by applicant]
CN 113316795A · 2021 [cited by applicant]
CN 113408634A · 2021 [cited by applicant]
CN 110263628B · 2021 [cited by applicant]
CN 114330714A · 2022 [cited by applicant]
CN 114330714B · 2022 [cited by applicant]
KR 100926876B1 · 2009 [cited by applicant]
KR 101242983B1 · 2013 [cited by applicant]
KR 101948089B1 · 2019 [cited by applicant]
KR 1020190119219A · 2019 [cited by applicant]
KR 102102772B1 · 2020 [cited by applicant]
KR 1020200132627A · 2020 [cited by applicant]
KR 1020210031627A · 2021 [cited by applicant]
KR 1020210045274A · 2021 [cited by applicant]
KR 1020210045845A · 2021 [cited by applicant]
KR 1020210053020A · 2021 [cited by applicant]
KR 1020210072559A · 2021 [cited by applicant]
KR 1020210076641A · 2021 [cited by applicant]
KR 102283523B1 · 2021 [cited by applicant]
KR 1020210092575A · 2021 [cited by applicant]
KR 102294714B1 · 2021 [cited by applicant]
KR 102310187B1 · 2021 [cited by applicant]
KR 102315617B1 · 2021 [cited by applicant]
KR 1020210141252A · 2021 [cited by applicant]
KR 1020210142829A · 2021 [cited by applicant]
KR 1020220000039A · 2022 [cited by applicant]
KR 1020220002046A · 2022 [cited by applicant]
KR 102422774B1 · 2022 [cited by applicant]
WO 2019108923A1 · 2019 [cited by applicant]
WO 2021042828A1 · 2021 [cited by applicant]
Cai et al., “Pruning the Unimportant or Redundant Filters? Synergy Makes Better,” 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China, 2021, pp. 1-8 (Year: 2021). [cited by examiner]
Sun et al., “Pruning Filters With L1-norm And Standard Deviation for CNN Compression,” Proc. of SPIE vol. 11041, 110412J (2019) (Year: 2019). [cited by examiner]
Li et al., “Pruning Filters for Efficient ConvNets,” arXiv:1608.08710v3 [cs.CV] Mar. 10, 2017 (Year: 2017). [cited by examiner]
Liu et al., “Linear Substitution Pruning: Consider All Filters Together,” ICCAI '22, Mar. 18-21, 2022, Tianjin, China (Year: 2022). [cited by examiner]
Lee et al., “SNIP: Single-shot Network Pruning based on Connection Sensitivity,” arXiv:1810.02340v2 [cs.CV] Feb. 23, 2019 (Year: 2019). [cited by examiner]
Choudhary, Tejalal et al. (2021). “A transfer learning with structured filter pruning approach for improved breast cancer classification on point-of-care devices”, Computers in Biology and Medicine. vol. 134, 1-13. Scie… [cited by applicant]
Lee, Jinsu et al. (2021). “An Overview of Energy-Efficient Hardware Accelerators for On-Device Deep-Neural-Network Training”, Open Journal of the Solid-State Circuits Society. vol. 1, 115-128. IEEE, doi: 10.1109/OJSSCS.… [cited by applicant]
LIm, Seung-Ho et al. (2022). “Architecture Exploration and Customization Tool of Deep Neural Networks for Edge Devices”, International Conference on Consumer Electronics (ICCE). 1-2. IEEE, doi: 10.1109/ICCE53296.2022.97… [cited by applicant]
Lu, Bingqian et al. (2019). “Automating Deep Neural Network Model Selection for Edge Inference”, First International Conference on Cognitive Machine Intelligence (CogMI). 184-193. IEEE, doi: 10.1109/CogMI48466.2019.0003… [cited by applicant]
Guo, Peizhen et al. (2021). “Mistify: Automating DNN Model Porting for On-Device Inference at the Edge”, 18th USENIX Symposium on Networked Systems Design and Implementation. 705-719. USENIX Association, URL:https://www… [cited by applicant]
Sharma, Hardik et al. (2016). “From High-Level Deep Neural Models to FPGAs”, 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 1-2. IEEE, doi:10.1109/MICRO.2016.7783720. [cited by applicant]
Lu, Bingqian et al. (2021). “One Proxy Device is Enough for Hardware-Aware Neural Architecture Search”, Proceedings of the ACM on Measurement and Analysis of Computing Systems. 1-34, 5(3) Article 34. Association for Com… [cited by applicant]
Hoefler, Torsten et al. (2021). “Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks”. 1-24, 22(1) Article 241. The Journal of Machine Learning Research, doi:10.5555/354… [cited by applicant]
Yeager, Luke et al. (2015). “DIGITS: the Deep learning GPU Training System”, ICML 2015 AutoML Workshop. 1-4. URL:https://indico.ijclab.in2p3.fr/event/2914/contributions/6476/subcontributions/170/attachments/6036/7161/di… [cited by applicant]
Kundu, Souvik et al. (2021). “Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression”. alarXiv:2107.12445v1 [cs.NE]. arXiv e-prints, doi: 10.48550/arXiv.2107.12445. [cited by applicant]
Cai, Han et al. (2018) “ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware”, arXiv preprint, arXiv:1812.00332v2 [cs.LG]. [cited by applicant]
Johnson, Jeff (1992) “Selectors: Going Beyond User-Interface Widgets”, Association for Computing Machinery, pp. 273-279. doi: 10.1145/142750.142810. [cited by applicant]
Decision to Grant a Patent for JP 2023-541299 by Japan Patent Office dated Feb. 6, 2024. [cited by applicant]
Decision to Grant a Patent for JP 2023-541308 by Japan Patent Office dated Feb. 6, 2024. [cited by applicant]
Office Action for U.S. Appl. No. 18/163,242 by United States Patent and Trademark Office dated Jan. 17, 2024. [cited by applicant]
International Search Report for PCT/KR2023/001895 by Korean Intellectual Property Office dated Apr. 20, 2023. [cited by applicant]
Luo, Jian-Hao et al. (2017). “ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression”, International Conference on Computer Vision (ICCV). arXiv:1707.06342v1 [cs.CV], 5068-5076. IEEE, doi: 10.1109/CCV… [cited by applicant]
Weng, Yui-Kai et al. (2021). “Block-Based Compression and Corresponding Hardware Circuits for Sparse Activations”, Sensors. 21(22), 7468, doi: 10.3390/s21227468. [cited by applicant]
Notice of Allowance and Fee(S) Due for U.S. Appl. No. 18/163,527 by United States Patent and Trademark Office dated May 30, 2023. [cited by applicant]
Grant of Patent for JP 2023-076065 by Japan Patent Office dated Jun. 6, 2020. [cited by applicant]
Zhu, Hongyu et al. (2018). “TBD: Benchmarking and Analyzing Deep Neural Network Training”, arXiv preprint, arXiv:1803.06905v2. [cited by applicant]
Office Action for KR 10-2023-0074346 by Korean Intellectual Property Office dated Aug. 22, 2023. [cited by applicant]
Office Action for U.S. Appl. No. 18/163,242 by United States Patent and Trademark Office dated Aug. 17, 2023. [cited by applicant]
Venieris, Stylianos I. et al. (2021). “OODIn: An Optimised On-Device Inference Framework for Heterogeneous Mobile Devices”, IEEE International Conference on Smart Computing (SMARTCOMP). arXiv:2106.04723v1 [cs.LG]. [cited by applicant]