IP Library › Granted Patent US 12,462,157
Granted Patent B2
US 12,462,157 · App. 17/846,555 · Granted Nov 4, 2025

Automatic channel pruning via graph neural network based hypernetwork

Inventors: Baopu Li (Santa Clara, CA); Qiuling Suo (Sunnyvale, CA); Yuchen Bian (Santa Clara, CA)
Assignee: Baidu USA LLC
G06N3/082G06N3/04G06N3/0499G06N3/0985G06N3/0455
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,462,157
App. No.
17/846,555
Granted
Nov 4, 2025
Kind
B2
Abstract

Model pruning is used to trim large neural networks, like convolutional neural networks (CNNs), to reduce computation overheads. Existing model pruning methods mainly rely on heuristics rules or local relationships of CNN layers. A novel hypernetwork based on graph neural network is disclosed for generating and evaluating pruned networks. A graph is first constructed according to information flow of channels and layers in a CNN network, with channels and layers represented as nodes and information flows represented as edges. A graph neural network is applied to aggregate both local and global dependencies across all channels and layers of the CNN network, resulting in informative node embeddings. With such embeddings, pruned CNN networks including their architectures and weights may be effectively generated and evaluated.

Claims (48)

1 . A computer-implemented method for using a neural network pruned via a graph neural network based hypernetwork, the method comprising:

providing input data to a final pruned neural network, the final pruned neural network being pruned via a graph neural network based hypernetwork pruning process, the graph neural network based hypernetwork pruning process comprising:

obtaining a pretrained neural network;

constructing an information flow graph based upon the pretrained neural network, the information flow graph comprising a node set and an edge set;

initializing node embeddings for nodes of the information flow graph;

responsive to a stop condition not being satisfied, performing steps associated with a graph neural network based hypernetwork, the steps comprising:

updating the node embeddings by applying a graph neural network to the information flow graph;

determining importance vectors of the pretrained neural network by applying a transformation function to at least some of the updated node embeddings;

determining mask indicators for channels of the pretrained neural network by applying normalization and/or binarization operations to the importance vectors, the mask indicators indicating which of the channels of the pretrained neural network to prune to obtain a pruned neural network;

determining weights for the pruned neural network by applying a feed-forward neural network to a function of the updated embeddings and corresponding importance vectors for the pruned neural network;

calculating a loss using a training data set; and

updating hypernetwork weights for the graph neural network based hypernetwork;

responsive to the stop condition being satisfied, providing one or more output pruned neural networks; and

selecting the final pruned neural network from the one or more output pruned neural networks; and

obtaining one or more output labels generated by the final pruned neural network in response to the input data.

2 . The computer-implemented method of claim 1 , wherein the node set of the information flow graph comprises one or more of:

layer nodes and channel nodes.

3 . The computer-implemented method of claim 1 , wherein initialized node embeddings for nodes of the information flow graph comprise randomly assigned embeddings or flattened weights of the pretrained neural network.

4 . The computer-implemented method of claim 3 , wherein the graph neural network comprises a graph convolutional network, a graph attention network, or a GraphSAGE network.

5 . The computer-implemented method of claim 1 , wherein the transformation function comprises a hyperbolic tangent of a function of at least the updated node embeddings and one or more learnable parameters.

6 . The computer-implemented method of claim 1 , wherein the feed-forward neural network comprises a multilayer perceptron for determining the weights from the at least some of the updated node embeddings and the at least some of the importance vectors, and wherein determining the weights for the pruned neural network comprises reshaping output of the multilayer perceptron into dimensions of the pruned neural network.

7 . The computer-implemented method of claim 1 , wherein the loss represents a cross-entropy loss for the pruned neural network and a compressionability loss.

8 . The computer-implemented method of claim 7 , wherein the compressionability loss is based upon a predefined floating-point operations per second (FLOPs) pruning ratio.

9 . The computer-implemented method of claim 1 , further comprising:

retraining or further training the final pruned neural network using the training data set.

10 . The computer-implemented method of claim 1 , wherein the pruning of the final pruned neural network via the graph neural network based hypernetwork pruning process contributes to a reduced inference time for generating the one or more output labels by the final pruned neural network in response to the input data relative to an inference time associated with generating one or more corresponding output labels by the pretrained neural network in response to the input data.

11 . The computer-implemented method of claim 1 , wherein the pretrained neural network comprises a pretrained convolutional neural network or a pretrained transformer neural network.

12 . The computer-implemented method of claim 1 , wherein the hypernetwork weights comprise weights for the graph neural network, weights for the transformation function, and weights for the feed-forward neural network.

13 . The computer-implemented method of claim 1 , wherein the stop condition comprises performance of a predetermined number of iterations, and wherein the graph neural network based hypernetwork pruning process further comprises, for at least some of the predetermined number of iterations, measuring an accuracy of the pruned neural network using a set of validation data.

14 . The computer-implemented method of claim 1 , wherein the final pruned neural network is selected from the one or more output pruned neural networks based upon measured accuracy, and wherein the graph neural network based hypernetwork pruning process further comprises retraining or further training the final pruned neural network using the training data set.

15 . A system configured for pruning a neural network via a graph neural network based hypernetwork, the system comprising:

one or more processors;

and one or more computer-readable media that store instructions that are executable by the one or more processors to configure the system to:

provide input data to a final pruned neural network, the final pruned neural network being pruned via a graph neural network based hypernetwork pruning process, and to:

obtain a pretrained neural network;

the graph neural network based hypernetwork pruning process comprising:

construct an information flow graph based upon the pretrained neural network, the information flow graph comprising a node set and an edge set;

initialize node embeddings for nodes of the information flow graph;

responsive to a stop condition not being satisfied, performing steps associated with a graph neural network based hypernetwork, the steps comprising:

updating the node embeddings by applying a graph neural network to the information flow graph;

determining importance vectors of the pretrained neural network by applying a transformation function to at least some of the updated node embeddings;

determining mask indicators for channels of the pretrained neural network by applying normalization and/or binarization operations to the importance vectors, the mask indicators indicating which of the channels of the pretrained neural network to prune to obtain a pruned neural network;

determining weights for the pruned neural network by applying a feed-forward neural network to a function of the updated embeddings and corresponding importance vectors for the pruned neural network;

calculating a loss using a training data set; and

updating hypernetwork weights for the graph neural network based hypernetwork; and

responsive to the stop condition being satisfied, providing one or more output pruned neural networks; and

selecting the final pruned neural network from the one or more output pruned neural networks; and

obtaining one or more output labels generated by the final pruned neural network in response to the input data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 21, 2022
From: LI, BAOPU; SUO, QIULING; BIAN, YUCHEN
To: BAIDU USA LLC
Reel/Frame 062174/0147 →
Continuity (2)
Provisional Application 63241095 · Sep 6, 2021
Related Publication 20230084203A1 · Mar 16, 2023
References Cited (51)
US 20190251441A1 · Lu · 2019 [cited by examiner]
US 20200372355A1 · Tarlow · 2020 [cited by examiner]
US 20210264278A1 · Liu · 2021 [cited by examiner]
Liu, Wenfeng, et al. “Locality Preserving Dense Graph Convolutional Networks with Graph Context-Aware Node Representations.” arXiv preprint arXiv:2010.05404 (2020). (Year: 2020). [cited by examiner]
Liu, Na, et al. “Stack attention-pruning aggregates multiscale graph convolution networks for hyperspectral remote sensing image classification.” IEEE Access 9 (2021): 44974-44988. (Year: 2021). [cited by examiner]
Xie, Zhouyang, et al. “Pruning with compensation: efficient channel pruning for deep convolutional neural networks.” arXiv preprint arXiv:2108.13728 (2021). (Year: 2021). [cited by examiner]
Wu et al.,“Causal Distillation for Language Models,” arXiv preprint arXiv:2112.02505, 2021. (7pgs). [cited by applicant]
Yang et al.,“Hierarchical Self-supervised Augmented Knowledge Distillation,” in Proceedings of International Joint Conference on Artificial Intelligence, 2021. (7pgs). [cited by applicant]
Yin et al.,“Towards efficient tensor decomposition-based dnn model compression with optimization framework,” arXiv preprint arXiv:2107.12422, 2021. (10pgs). [cited by applicant]
Zeiler et al.,“Visualizing and understanding convolutional networks,” arXiv preprint arXiv:1311.2901, 2013. (11pgs). [cited by applicant]
Zhang et al.,“Efficient inference of CNNs via channel pruning,” arXiv preprint arXiv:1908.03266, 2019. (8pgs). [cited by applicant]
Zhang et al.,“Graph HyperNetworks for Neural Architecture Search,” arXiv preprint arXiv:1810.05749, 2018. (16pgs). [cited by applicant]
Zhang et al.,“Graph Pruning for Model Compression,” arXiv preprint arXiv:1911.09817, 2021. (12pgs). [cited by applicant]
Zhao et al.,“Variational Convolutional Neural Network Pruning,” in Proceedings of the IEEE/CVF Conference on Computer Vision & Pattern Recognition, 2019. (10pgs). [cited by applicant]
Zhuang et al.,“Discrimination-aware channel pruning for deep neural networks,” Advances in neural information processing systems, 2018. (12pgs). [cited by applicant]
Hamilton et al.,“Inductive representation learning on large graphs,” Advances in neural information processing systems, 2017. (11pgs). [cited by applicant]
He et al.,“Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks,” arXiv preprint arXiv:1808.06866, 2018. (8pgs). [cited by applicant]
He et al.,“AMC: AutoML for Model Compression and Acceleration on Mobile Devices,” in Proceedings of the European conference on computer vision (ECCV), 2018. (17pgs). [cited by applicant]
He et al.,“Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration,”, in Proceedings of the IEEE/CVF Conference on Computer Vision & Pattern Recognition, 2019. (10pgs). [cited by applicant]
He et al.,“Channel Pruning for Accelerating Very Deep Neural Networks,” in Proceedings of the IEEE international conference on computer vision, 2017. (9pgs). [cited by applicant]
Huang et al.,“Data-Driven Sparse Structure Selection for Deep Neural Networks,” in Proceedings of the European conference on computer vision (ECCV), 2018. (17pgs). [cited by applicant]
Kipf et al.,“Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2017. (14pgs). [cited by applicant]
Koratana et al.,“LIT: Learned Intermediate Representation Training for Model Compression,” in Proceedings of the International Conference on Machine Learning, 2019. (10pgs). [cited by applicant]
Le et al.,“Tiny ImageNet Visual Recognition Challenge,” CS231N course, Stanford University, 2015. (6pgs). [cited by applicant]
Bengio et al.,“Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432, 2013. (12pgs). [cited by applicant]
Brock et al.,“SMASH: One-Shot Model Architecture Search through HyperNetworks,” in Proc. of the International Conference on Learning Representations, 2018. (22pgs). [cited by applicant]
Chen et al.,“A Hierarchical Graph Network for 3D Object Detection on Point Clouds,” in Proc. of the IEEE/CVF conf. on computer vision & pattern recognition, 2020. (10pgs). [cited by applicant]
Ding et al.,“Approximated Oracle Filter Pruning for Destructive CNN Width Optimization,” in Proc. of the International Conference on Machine Learning, 2019. (10pgs). [cited by applicant]
Dong et al.,“Network Pruning via Transformable Architecture Search,” Advances in Neural Information Processing Systems, 2019. (12pgs). [cited by applicant]
Gao et al.,“Discrete Model Compression With Resource Constraint for Deep Neural Networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020. (10pgs). [cited by applicant]
Guo et al.,“Channel Pruning Guided by Classification Loss and Feature Importance,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2020. (8pgs). [cited by applicant]
Guo et al.,“DMCP: Differentiable Markov Channel Pruning for Neural Networks,” in Proceedings of the IEEE/CVF conference on computer vision & pattern recognition, 2020. (9pgs). [cited by applicant]
Ha et al.,“HyperNetworks,” in Proceedings of the International Conference on Learning Representations, 2017. (18pgs). [cited by applicant]
Lu et al.,“Cross-Modality Person Re-Identification With Shared-Specific Feature Transfer,” in Proceedings of the IEEE/CVF Conference on Computer Vision & Pattern Recognition (CVPR), 2020. (11pgs). [cited by applicant]
Ning et al.,“DSA: More efficient budgeted pruning via differentiable sparsity allocation,” in Proc. of the European Conference on Computer Vision, Springer, 2020. (16pgs). [cited by applicant]
Siddegowda et al.,“Neural Network Quantization with AI Model Efficiency Toolkit (AIMET),” arXiv preprint arXiv:2201.08442, 2022. (39pgs). [cited by applicant]
Tay et al.,“HyperGrid Transformers: Towards A Single Model for Multiple Tasks,” in Proceedings of the International Conference on Learning Representations, 2021. (14 pgs). [cited by applicant]
Veličković et al.,“Graph Attention Networks,” arXiv preprint arXiv:1710.10903, 2018. (12pgs). [cited by applicant]
Johannes von Oswald et al.,“Continual learning with hypernetworks,” arXiv preprint arXiv:1906.00695, 2019. (16pgs). [cited by applicant]
Wang et al.,“Fully quantized image super-resolution networks,” arXiv preprint arXiv:2011.14265, 2021. (12pgs). [cited by applicant]
Wang et al.,“Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks,” arXiv preprint arXiv:1811.08589, 2018. (7pgs). [cited by applicant]
Wang et al.,“Post-OCR Paragraph Recognition by Graph Convolutional Networks,” In Proc. of the IEEE/CVF Winter Conf. on Applications of Computer Vision, 2022. (10pgs). [cited by applicant]
Li et al.,“Automatic Channel Pruning with Hyper-parameter Search and Dynamic Masking,” in Proceedings of the ACM International Conference on Multimedia, 2021. (9pgs). [cited by applicant]
Li et al.,“Pruning Filters for Efficient ConvNets,” arXiv preprint arXiv:1608.08710, 2016. (9pgs). [cited by applicant]
Li et al.,“HTD: Heterogeneous Task Decoupling for Two-Stage Object Detection,” IEEE Transactions on Image Processing, 2021. (14pgs). [cited by applicant]
Li et al.,“Group Sparsity: The Hinge Between Filter Pruning & Decomposition for Network Compression,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020. (10pgs). [cited by applicant]
Li et al.,“DHP: Differentiable Meta Pruning via HyperNetworks,” in Proceedings of the European Conference on Computer Vision, 2020. (17pgs). [cited by applicant]
Li et al.,“Adaptive Feature Fusion via Graph Neural Network for Person Re-identification,” in Proceedings of the ACM International Conference on Multimedia, 2019. (9pgs). [cited by applicant]
Lin et al.,“HRank: Filter Pruning using High-Rank Feature Map,” in CVPR, 2020. (10pgs). [cited by applicant]
Liu et al.,“Learning Efficient Convolutional Networks through Network Slimming,” in Proceedings of the IEEE International Conference on Computer Vision, 2017. (9pgs). [cited by applicant]
Liu et al.,“MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning,” in Proc. of the IEEE/CVF international conf. on computer vision, 2019. (10pgs). [cited by applicant]