IP Library › Granted Patent US 12,566,946
Granted Patent B2
US 12,566,946 · App. 17/615,077 · Granted Mar 3, 2026

Sparse convolutional neural network

Inventors: Chen Zhang (Beijing, CN); Yunxin Liu (Beijing, CN)
Assignee: Microsoft Technology Licensing, LLC
G06N3/048G06F5/01G06F17/15G06N3/0495
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,566,946
App. No.
17/615,077
Granted
Mar 3, 2026
Kind
B2
Abstract

Various implementations of the subject matter as described herein relate to a sparse convolutional neural network. In some implementations, a computer-implemented method comprises: quantizing an input feature map to obtain a quantized input feature map; determining, based on the quantized input feature map, a sparsity mask for an output feature map through a quantized version of a convolutional neural network, the sparsity mask indicating positions of non-zero entries in the output feature map; and determining, based on the input feature map, the non-zero entries indicated by the sparsity mask in the output feature map through the convolutional neural network.

Claims (57)

1 . A computer-implemented method, comprising:

transforming an input into an input feature map via a convolutional neural network executing on a computer;

quantizing the input feature map to obtain a quantized input feature map;

determining, based on the quantized input feature map, a sparsity mask for an output feature map via a prediction module using a quantized version of the convolutional neural network and a quantized convolution layer, the sparsity mask indicating positions of non-zero entries in the output feature map;

determining, via a sparse convolution layer and based on the input feature map, the non-zero entries indicated by the sparsity mask in the output feature map via an activation function of the convolutional neural network; and

providing a categorized result via the convolutional neural network in response to the input using the non-zero entries to perform convolutional neural network calculations and using zero entries of the output feature map to skip convolutional neural network calculations.

2 . The method of claim 1 , wherein quantizing the input feature map comprises:

quantizing the input feature map in no more than 4 bits.

3 . The method of claim 1 , wherein determining the sparsity mask comprises:

determining, based on the quantized input feature map, a quantized output feature map through a quantized convolution operation; and

determining, based on the quantized output feature map, the sparsity mask.

4 . The method of claim 1 , wherein determining the sparsity mask comprises:

determining, based on the quantized input feature map, a quantized output feature map through a quantized convolution and batch normalization operation, the convolution and batch normalization operation being fusion of a convolution operation and a batch normalization operation; and

determining, based on the quantized output feature map, the sparsity mask.

5 . The method of claim 3 , where determining, based on the quantized output feature map, the sparsity mask comprises:

providing the quantized output feature map to an activation function, to determine an output of the activation function; and

determining, based on a sign output by the activation function, the sparsity mask.

6 . The method of claim 1 , wherein the sparsity mask is encoded to record indices of non-entries only.

7 . The method of claim 1 , wherein the method is implemented by a general purpose processor, and wherein the quantizing is performed by a vector processing unit of the general purpose processor.

8 . The method of claim 7 , wherein data format transformation between the vector processing unit and a kernel of the general purpose processor is implemented by an in-register shift operation.

9 . The method of claim 1 , wherein the method is implemented by a hardware enabling a mixed precision arithmetic operation.

10 . A device, comprising:

a processing unit; and

a memory coupled to the processing unit and having instructions stored thereon, the instructions, when executed by the processing unit, causing the device to perform acts comprising:

transforming an input into an input feature map via a convolutional neural network executing on a computer;

quantizing the input feature map to obtain a quantized input feature map;

determining, based on the quantized input feature map, a sparsity mask for an output feature map via a prediction module using a quantized version of the convolutional neural network and a quantized convolution layer, the sparsity mask indicating positions of non-zero entries in the output feature map;

determining, via a sparse convolution layer and based on the input feature map, the non-zero entries indicated by the sparsity mask in the output feature map via an activation function of the convolutional neural network; and

providing a categorized result via the convolutional neural network in response to the input using the non-zero entries to perform convolutional neural network calculations and using zero entries of the output feature map to skip convolutional neural network calculations.

11 . The device of claim 10 , wherein quantizing the input feature map comprises:

quantizing the input feature map in no more than 4 bits.

12 . The device of claim 10 , wherein determining the sparsity mask comprises:

determining, based on the quantized input feature map, a quantized output feature map through a quantized convolution operation; and

determining, based on the quantized output feature map, the sparsity mask.

13 . The device of claim 10 , wherein determining the sparsity mask comprises:

determining, based on the quantized input feature map, a quantized output feature map through a quantized convolution and batch normalization operation, the convolution and batch normalization operation being fusion of a convolution operation and a batch normalization operation; and

determining, based on the quantized output feature map, the sparsity mask.

14 . The device of claim 12 , where determining, based on the quantized output feature map, the sparsity mask comprises:

providing the quantized output feature map to an activation function, to determine an output of the activation function; and

determining, based on a sign output by the activation function, the sparsity mask.

15 . A computer program product stored in a computer storage medium and comprising computer-executable instructions, the computer-executable instructions, when executed by a device, causing the device to perform acts comprising:

transforming an input into an input feature map via a convolutional neural network executing on a computer;

quantizing the input feature map to obtain a quantized input feature map;

determining, based on the quantized input feature map, a sparsity mask for an output feature map via a prediction module using a quantized version of the convolutional neural network and a quantized convolution layer, the sparsity mask indicating positions of non-zero entries in the output feature map;

determining, via a sparse convolution layer and based on the input feature map, the non-zero entries indicated by the sparsity mask in the output feature map via an activation function of the convolutional neural network; and

providing a categorized result via the convolutional neural network in response to the input using the non-zero entries to perform convolutional neural network calculations and using zero entries of the output feature map to skip convolutional neural network calculations.

16 . The computer program product of claim 15 , wherein quantizing the input feature map comprises quantizing the input feature map in no more than 4 bits.

17 . The computer program product of claim 15 , wherein determining the sparsity mask comprises:

determining, based on the quantized input feature map, a quantized output feature map through a quantized convolution operation; and

determining, based on the quantized output feature map, the sparsity mask.

18 . The computer program product of claim 15 , wherein determining the sparsity mask comprises:

determining, based on the quantized input feature map, a quantized output feature map through a quantized convolution and batch normalization operation, the convolution and batch normalization operation being fusion of a convolution operation and a batch normalization operation; and

determining, based on the quantized output feature map, the sparsity mask.

19 . The computer program product of claim 15 , where determining, based on the quantized output feature map, the sparsity mask comprises:

providing the quantized output feature map to an activation function, to determine an output of the activation function; and

determining, based on a sign output by the activation function, the sparsity mask.

20 . The computer program product of claim 15 , wherein the sparsity mask is encoded to record indices of non-entries only.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 6, 2021
From: ZHANG, CHEN; LIU, YUNXIN
To: MICROSOFT TECHNOLOGY LICENSING, LLC
Reel/Frame 058291/0690 →
Priority Claims (1)
CN 201910527910.8 · Jun 18, 2019 · national
Continuity (1)
Related Publication 20220245433A1 · Aug 4, 2022
References Cited (50)
US 10332001B2 · Rippel · 2019 [cited by examiner]
US 10832123B2 · Sun · 2020 [cited by examiner]
US 20200143250A1 · Lee · 2020 [cited by examiner]
US 20220245433A1 · Zhang et al. · 2022 [cited by applicant]
WO 2018073975A1 · 2018 [cited by applicant]
WO 2018214913A1 · 2018 [cited by applicant]
CVPR2019, program, main conference, 2019, found at https://cvpr2019.thecvf.com/program/main_conference (Year: 2019). [cited by examiner]
Notice of Allowance Received for Chinese Application No. 201910527910.8, mailed on Dec. 11, 2024, 04 pages. (English Translation Provided). [cited by applicant]
First Office Action Received for Chinese Application No. 201910527910.8, mailed on Jun. 26, 2024, 13 pages. (English Translation Provided). [cited by applicant]
“Nvidia Turing GPU Architecture”, https://images.nvidia.com/aem-dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf, May 15, 2021, 86 Pages. [cited by applicant]
Krizhevsky, et al., “The CIFAR-10 Dataset”, https://web.archive.org/web/20090923012829/https://www.cs.toronto.edu/˜kriz/cifar.html, Sep. 23, 2009, 4 Pages. [cited by applicant]
Akhlaghi, et al., “Snapea: Predictive Early Activation for Reducing Computation in Deep Convolutional Neural Networks”, In Proceedings of the 45th International Symposium on Computer Architecture (ISCA), Jun. 1, 2018, 1… [cited by applicant]
Cao, et al., “SeerNet: Predicting Convolutional Neural Network Feature-Map Sparsity Through Low-Bit Quantization”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 15, 2019, pp. 112… [cited by applicant]
Dong, et al., “More is less: A More Complicated Network with Less Inference Complexity”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 21, 2017, pp. 1895-1903. [cited by applicant]
Figurnov, et al., “Spatially Adaptive Computation Time for Residual Networks”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jul. 21, 2017, pp. 1039-1048. [cited by applicant]
Graham, et al., “Submanifold Sparse Convolutional Networks”, In Repository of arXiv:1706.01307v1, Jun. 5, 2017, 10 Pages. [cited by applicant]
Han, et al., “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding”, In Repository of arXiv:1510.00149v1, Oct. 1, 2015, 11 Pages. [cited by applicant]
Han, et al., “Learning Both Weights and Connections for Efficient Neural Networks”, In Proceedings of Advances in Neural Information Processing Systems, vol. 1, Jun. 8, 2015, 9 Pages. [cited by applicant]
He, et al., “Deep Residual Learning for Image Recognition”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 27, 2016, pp. 770-778. [cited by applicant]
Iandola, et al., “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size”, In Repository of arXiv:1602.07360v4, Nov. 4, 2016, 13 Pages. [cited by applicant]
Ioffe, et al., “Batch normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, In Repository of arxiv:1502.03167v1, Feb. 11, 2015, 10 Pages. [cited by applicant]
Jaderberg, et al., “Speeding up Convolutional Neural Networks with Low Rank Expansions”, In Repository of arXiv:1405.3866v1 , May 15, 2014, 12 Pages. [cited by applicant]
Jaderberg, et al., “Speeding up Convolutional Neural Networks with Low Rank Expansions”, In Proceedings of British Machine Vision Conference, Sep. 1, 2014, 13 Pages. [cited by applicant]
Judd, et al., “Cnvlutin2: Ineffectual-Activation-and-Weight-Free Deep Neural Network Computing”, In Repository of arXiv:1705.00125v1, Apr. 29, 2017, 6 Pages. [cited by applicant]
Kong, et al., “RON: Reverse Connection with Objectness Prior Networks for Object Detection”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 21, 2017, pp. 5244-5252. [cited by applicant]
Krishnamoorthi, Raghuraman, “Quantizing Deep Convolutional Networks for Efficient Inference: A whitepaper”, In Repository of arXiv:1806.08342v1 , Jun. 21, 2018, 36 Pages. [cited by applicant]
Li, et al., “Not All Pixels Are Equal: Difficulty-Aware Semantic Segmentation via Deep Layer Cascade”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 21, 2017, pp. 6459-6468. [cited by applicant]
Li, et al., “Pruning Filters for Efficient ConvNets”, In Repository of arXiv:1608.08710v1, Aug. 31, 2016, 9 Pages. [cited by applicant]
Liu, et al., “Sparse Convolutional Neural Networks”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 7, 2015, pp. 806-814. [cited by applicant]
Loannou, et al., “Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 21, 2017, pp. 5977-5986. [cited by applicant]
Mcdanel, et al., “Full-Stack Optimization for Accelerating CNNs with FPGA Validation”, In Repository of arXiv:1905.00462v1 May 1, 2019, 12 Pages. [cited by applicant]
Parashar, et al., “SCNN: An Accelerator for Compressed-Sparse Convolutional Neural Networks”, In Proceedings of ACM SIGARCH Computer Architecture News vol. 45, Issue 2, Jun. 24, 2017, pp. 27-40. [cited by applicant]
“International Search Report & Written Opinion issued in PCT Application No. PCT/US20/030327”, Mailed Date: Jul. 30, 2020, 13 Pages. [cited by applicant]
Rastegari, et al., “XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks”, In Proceedings of European Conference on Computer Vision, 2016, pp. 525-542. [cited by applicant]
Ren, et al., “SBNet: Sparse Blocks Network for Fast Inference”, In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 18, 2018, pp. 8711-8720. [cited by applicant]
Russakovsky, et al., “ImageNet Large Scale Visual Recognition Challenge”, In International Journal of Computer Vision, vol. 115, No. 3, Apr. 11, 2015, pp. 211-252. [cited by applicant]
Shi, et al., “PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud”, In Repository of arXiv:1812.04244v2, Dec. 11, 2018, 10 Pages. [cited by applicant]
Shi, et al., “Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units”, In Repository of arXiv:1704.07724v1, Apr. 25, 2017, 7 Pages. [cited by applicant]
Zhou, et al., “Dorefa-net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients”, In Repository of arXiv:1606.06160v1, Jun. 20, 2016, 13 Pages. [cited by applicant]
Simonyan, et al., “Very Deep Convolutional Networks for Large-Scale Image Recognition”, In Repository of arxiv:1409.1556v1, Sep. 4, 2014, 10 Pages. [cited by applicant]
Szegedy, et al., “Going Deeper with Convolutions”, In Proceedings of the IEEE conference on computer vision and pattern recognition, Jun. 7, 2015, 9 pages. [cited by applicant]
Szegedy, et al., “Rethinking the Inception Architecture for Computer Vision”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 27, 2016, pp. 2818-2826. [cited by applicant]
Vanhoucke, et al., “Improving the Speed of Neural Networks on CPUs”, In Proceedings of Deep Learning and Unsupervised Feature Learning Workshop, NIPS, 2011, 8 Pages. [cited by applicant]
Wen, et al., “Learning Structured Sparsity in Deep Neural Networks”, In Proceedings of Advances in Neural Information Processing Systems, Dec. 2016, 9 Pages. [cited by applicant]
Wu, et al., “3d Shapenets: A Deep Representation for Volumetric Shapes”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 7, 2015, pp. 1912-1920. [cited by applicant]
Yan, et al., “SECOND: Sparsely Embedded Convolutional Detection”, In Journals of Sensors, vol. 18, Issue 10, Oct. 6, 2018, 17 Pages. [cited by applicant]
Yao, et al., “Balanced Sparsity for Efficient DNN Inference on GPU”, In Repository of arXiv:1811.00206v1, Nov. 1, 2018, 8 Pages. [cited by applicant]
Zhang, et al., “LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks”, In Repository of arXiv:1807.10029v1, Jul. 26, 2018, 21 Pages. [cited by applicant]
Zhu, et al., “Trained Ternary Quantization”, In Repository of arXiv:1612.01064v1, Dec. 4, 2016, 9 Pages. [cited by applicant]
Zhuang, et al., “Towards Effective Low-bitwidth Convolutional Neural Networks”, In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 18, 2018, pp. 7920-7928. [cited by applicant]