IP Library › Granted Patent US 12,547,878
Granted Patent B2
US 12,547,878 · App. 18/486,534 · Granted Feb 10, 2026

Highly efficient convolutional neural networks

Inventors: Andrew Gerald Howard (Culver City, CA); Mark Sandler (Mountain View, CA); Liang-Chieh Chen (Los Angeles, CA); Andrey Zhmoginov (Mountain View, CA); Menglong Zhu (Playa Vista, CA)
Assignee: GOOGLE LLC
G06N3/04G06N3/045G06N3/08G06N3/048
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,547,878
App. No.
18/486,534
Granted
Feb 10, 2026
Kind
B2
Abstract

The present disclosure provides directed to new, more efficient neural network architectures. As one example, in some implementations, the neural network architectures of the present disclosure can include a linear bottleneck layer positioned structurally prior to and/or after one or more convolutional layers, such as, for example, one or more depthwise separable convolutional layers. As another example, in some implementations, the neural network architectures of the present disclosure can include one or more inverted residual blocks where the input and output of the inverted residual block are thin bottleneck layers, while an intermediate layer is an expanded representation. For example, the expanded representation can include one or more convolutional layers, such as, for example, one or more depthwise separable convolutional layers. A residual shortcut connection can exist between the thin bottleneck layers that play a role of an input and output of the inverted residual block.

Claims (58)

1 . A computing system comprising:

one or more processors; and

one or more non-transitory computer-readable media that store:

a convolutional neural network configured to process an input image to extract features from the input image, the convolutional neural network comprising:

a convolutional block comprising:

a first expansion layer configured to expand a first input feature representation with a first input number of channels to a first intermediate number of channels higher than the first input number of channels;

a first depthwise convolutional layer downstream of the first expansion layer that is configured to apply a first depthwise convolution to generate a first intermediate feature representation with the first intermediate number of channels;

a first pointwise convolutional layer downstream of the first depthwise convolutional layer that is configured to apply a first pointwise convolution to generate a first output feature representation, wherein the first output feature representation comprises features that are linearly projected from the first intermediate number of channels to a first output number of channels lower than the first intermediate number of channels; and

an inverted residual bottleneck block downstream of the convolutional block, a residual shortcut connection connecting an input and an output of the inverted residual bottleneck block, the inverted residual bottleneck block comprising:

a second expansion layer configured to expand a second input feature representation with a second input number of channels to a second intermediate number of channels higher than the second input number of channels;

a second depthwise convolutional layer downstream of the second expansion layer that is configured to apply a second depthwise convolution to generate a second intermediate feature representation with the second intermediate number of channels; and

a second pointwise convolutional layer downstream of the second depthwise convolutional layer that is configured to apply a second pointwise convolution to generate a second output feature representation, wherein the second output feature representation comprises features that are linearly projected from the second intermediate number of channels to a second output number of channels lower than the second intermediate number of channels.

2 . The computing system of claim 1 , wherein the inverted residual bottleneck block is ordered immediately subsequent to the convolutional block.

3 . The computing system of claim 2 , wherein the convolutional block comprises a residual shortcut connection connecting an input and an output of the convolutional block.

4 . The computing system of claim 1 , wherein the second output number of channels of the inverted residual bottleneck block comprises one-half as many channels as an output number of channels of a downstream inverted residual bottleneck block.

5 . The computing system of claim 1 , wherein the second intermediate number of channels is six times as many channels as the second input number of channels.

6 . The computing system of claim 1 , wherein the second depthwise convolutional layer is configured to apply a relu6 operator.

7 . The computing system of claim 1 , wherein the one or more non-transitory computer-readable media store:

a machine-learned model configured to process a feature map generated by the convolutional neural network.

8 . The computing system of claim 7 , wherein the machine-learned model is configured to perform an image processing task using the feature map.

9 . The computing system of claim 8 , wherein the image processing task comprises recognizing objects in the input image.

10 . The computing system of claim 1 , wherein the one or more non-transitory computer-readable media store instructions that are executable by the one or more processors to cause the computing system to perform operations comprising:

generating a feature map by processing an image using the convolutional neural network; and

processing the feature map using a machine-learned model to recognize objects in the image.

11 . The computing system of claim 10 , wherein the operations comprise:

receiving the image from a client computing device; and

returning an output generated using the convolutional neural network as part of a web service.

12 . The computing system of claim 10 , wherein the one or more non-transitory computer-readable media are included in a mobile computing device.

13 . The computing system of claim 1 , wherein the one or more non-transitory computer-readable media store instructions that are executable by the one or more processors to cause the computing system to perform operations comprising:

generating a feature map by processing an image using the convolutional neural network; and

processing the feature map to perform an image processing task.

14 . The computing system of claim 1 , wherein the first depthwise convolutional layer is configured to perform depthwise convolution with a stride of two.

15 . The computing system of claim 1 , wherein the input and the output of the inverted residual bottleneck block correspond to linear bottleneck layers.

16 . A computer-implemented method, comprising:

providing, by a computing system comprising one or more processors, an image as input to a convolutional neural network, the convolutional neural network comprising:

a convolutional block comprising:

a first expansion layer configured to expand a first input feature representation with a first input number of channels to a first intermediate number of channels higher than the first input number of channels;

a first depthwise convolutional layer downstream of the first expansion layer that is configured to apply a first depthwise convolution to generate a first intermediate feature representation with the first intermediate number of channels;

a first pointwise convolutional layer downstream of the first depthwise convolutional layer that is configured to apply a first pointwise convolution to generate a first output feature representation, wherein the first output feature representation comprises features that are linearly projected from the first intermediate number of channels to a first output number of channels lower than the first intermediate number of channels; and

an inverted residual bottleneck block downstream of the convolutional block, a residual shortcut connection connecting an input and an output of the inverted residual bottleneck block, the inverted residual bottleneck block comprising:

a second expansion layer configured to expand a second input feature representation with a second input number of channels to a second intermediate number of channels higher than the second input number of channels;

a second depthwise convolutional layer downstream of the second expansion layer that is configured to apply a second depthwise convolution to generate a second intermediate feature representation with the second intermediate number of channels; and

a second pointwise convolutional layer downstream of the second depthwise convolutional layer that is configured to apply a second pointwise convolution to generate a second output feature representation, wherein the second output feature representation comprises features that are linearly projected from the second intermediate number of channels to a second output number of channels lower than the second intermediate number of channels; and

generating, by the computing system and using the convolutional neural network, a feature map from the image.

17 . The computer-implemented method of claim 16 , comprising:

processing the feature map to perform an image processing task.

18 . The computer-implemented method of claim 17 , comprising:

processing, by the computing system and using a machine-learned model configured to perform the image processing task, the feature map.

19 . The computer-implemented method of claim 18 , wherein the image processing task comprises recognizing objects in the image or semantic segmentation of the image.

20 . One or more non-transitory computer-readable media that store a convolutional neural network configured to process an input image to extract features from the input image, the convolutional neural network comprising:

a convolutional block comprising:

a first expansion layer configured to expand a first input feature representation with a first input number of channels to a first intermediate number of channels higher than the first input number of channels;

a first depthwise convolutional layer downstream of the first expansion layer that is configured to apply a first depthwise convolution to generate a first intermediate feature representation with the first intermediate number of channels;

a first pointwise convolutional layer downstream of the first depthwise convolutional layer that is configured to apply a first pointwise convolution to generate a first output feature representation, wherein the first output feature representation comprises features that are linearly projected from the first intermediate number of channels to a first output number of channels lower than the first intermediate number of channels; and

an inverted residual bottleneck block downstream of the convolutional block, a residual shortcut connection connecting an input and an output of the inverted residual bottleneck block, the inverted residual bottleneck block comprising:

a second expansion layer configured to expand a second input feature representation with a second input number of channels to a second intermediate number of channels higher than the second input number of channels;

a second depthwise convolutional layer downstream of the second expansion layer that is configured to apply a second depthwise convolution to generate a second intermediate feature representation with the second intermediate number of channels; and

a second pointwise convolutional layer downstream of the second depthwise convolutional layer that is configured to apply a second pointwise convolution to generate a second output feature representation, wherein the second output feature representation comprises features that are linearly projected from the second intermediate number of channels to a second output number of channels lower than the second intermediate number of channels.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 2, 2023
From: HOWARD, ANDREW GERALD; SANDLER, MARK; CHEN, LIANG-CHIEH; ZHMOGINOV, ANDREY; ZHU, MENGLONG
To: GOOGLE LLC
Reel/Frame 065438/0916 →
Continuity (4)
Continuation 17382503 · Jul 22, 2021
Continuation 15898566 · Feb 17, 2018
Provisional Application 62586007 · Nov 14, 2017
Related Publication 20240119256A1 · Apr 11, 2024
References Cited (74)
US 11379985B2 · Wang et al. · 2022 [cited by applicant]
US 20150294219A1 · Krizhevsky · 2015 [cited by applicant]
US 20180240235A1 · Mazo · 2018 [cited by applicant]
US 20190179674A1 · Huang · 2019 [cited by examiner]
US 20240119256A1 · Howard et al. · 2024 [cited by applicant]
CN 106204467A · 2016 [cited by applicant]
CN 106845371A · 2017 [cited by applicant]
CN 107316079A · 2017 [cited by applicant]
Chollet, François. “Xception: Deep learning with depthwise separable convolutions.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. (Year: 2017). [cited by examiner]
Chen, Tiffany Yu-Han, et al. “Glimpse: Continuous, real-time object recognition on mobile devices.” Proceedings of the 13th ACM conference on embedded networked sensor systems. 2015. (Year: 2015). [cited by examiner]
Yu, Dong, et al. “Deep Convolutional Neural Networks with Layer-Wise Context Expansion and Attention.” Interspeech. 2016. (Year: 2016). [cited by examiner]
Pérez-Espinosa, Humberto, et al. “Tuning the parameters of a convolutional artificial neural network by using covering arrays.” Research in Computing Science 121 (2016): 69-81. (Year: 2016). [cited by examiner]
Sifre, Laurent, and Stéphane Mallat. “Rigid-motion scattering for texture classification.” arXiv preprint arXiv:1403.1687 (2014). (Year: 2014). [cited by examiner]
Chang, Jia-Ren, and Yong-Sheng Chen. “Deep Competitive Pathway Networks.” arXiv preprint arXiv:1709.10282 (2017). (Year: 2017). [cited by examiner]
Ghosh, Tapabrata. “Quicknet: maximizing efficiency and efficacy in deep architectures.” arXiv preprint arXiv:1701.02291 (2017). (Year: 2017). [cited by examiner]
Drozdzal, Michal, et al. “The importance of skip connections in biomedical image segmentation.” International Workshop on Deep Learning in Medical Image Analysis. Cham: Springer International Publishing, 2016. (Year: 20… [cited by examiner]
Rubin, Jonathan, et al. “Densely connected convolutional networks and signal quality analysis to detect atrial fibrillation using short single-lead ECG recordings.” 2017 Computing in cardiology (cinc). IEEE, 2017. (Year… [cited by examiner]
Deshpande, Adit. “A beginner's guide to understanding convolutional neural networks.” Retrieved Mar. 31, 2017 (2016). (Year: 2016). [cited by examiner]
Lin, Zhouhan, Roland Memisevic, and Kishore Konda. “How far can we go without convolution: Improving fully-connected networks.” arXiv preprint arXiv:1511.02580 (2015). (Year: 2015). [cited by examiner]
Chang, Jia-Ren, and Yong-Sheng Chen. “Deep competitive pathway networks.” Asian Conference on Machine Learning. PMLR, 2017. (Year: 2017). [cited by examiner]
Jégou, Simon, et al. “The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation.” Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017. (Year: 2017… [cited by examiner]
Machine Translated Chinese Search Report Corresponding to Application No. 2021108693785 on Jun. 25, 2024. [cited by applicant]
Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems”, arXiv: 1603.04467v2, Mar. 16, 2016, 19 pages. [cited by applicant]
Ahmed et al., “Connectivity Learning in Multi-Branch Networks”, arXiv:1709.09582v2, Dec. 7, 2017, 17 pages. [cited by applicant]
Bergstra et al., “Random Search for Hyper-Parameter Optimization”, Journal of Machine Learning Research, vol. 13, Feb. 2012, pp. 281-305. [cited by applicant]
Chang et al., “Deep Competitive Pathway Networks”, arXiv.org, Cornell University Library, Ithaca, New York, Sep. 29, 2017, 16 pages. [cited by applicant]
Changpinyo et al., “The Power of Sparsity in Convolutional Neural Networks”, arXiv: 1702.06257v1, Feb. 21, 2017, 13 pages. [cited by applicant]
Chen et al., “Deeplab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs”, arXiv: 1606.00915v2, May 12, 2017, 14 pages. [cited by applicant]
Chen et al., “Rethinking Atrous Convolution for Semantic Image Segmentation”, arXiv: 1706.05587v3, Dec. 5, 2017, 14 pages. [cited by applicant]
Chollet et al., Xception: Deep Learning with Depthwise Separable Convolutions, Apr. 4, 2017, https://arxiv.org/pdf/1610.02357.pdf, retrieved on Aug. 15, 2018. [cited by applicant]
International Search Report and Written Opinion for PCT/US2018/046022, mailed on Nov. 20, 2018, 16 pages. [cited by applicant]
Dai et al., “Object Detection via Region-Based Fully Convolutional Networks”, Advances in Neural Information Processing Systems, Barcelona, Spain, Dec. 5-10, 2016, pp. 379-387. [cited by applicant]
Esteban et al., “Large-Scale Evolution of Image Classifiers”, 34 [cited by applicant]
Everingham et al., “The Pascal Visual Object Classes Challenge a Retrospect”, International Journal on Computer Vision, vol. 111, Issue 1, Jan. 2015, pp. 98-136. [cited by applicant]
Guo et al., “Dynamic Network Surgery for Efficient DNNS”, Advances in Neural Information Processing Systems, Barcelona, Spain, Dec. 5-10, 2016, pp. 1379-1387. [cited by applicant]
Han et al., “DSD: Regularizing Deep Neural Networks with Dense-Sparse-Dense Training Flow”, arXiv: 1607.04381v1, Jul. 15, 2016, 12 pages. [cited by applicant]
Han et al., “Learning Both Weights and Connections for Efficient Neural Network”, Advances in Neural Information Processing Systems, Montreal, Quebec, Canada, Dec. 7-12, 2015, pp. 1135-1143. [cited by applicant]
Hariharan et al., “Semantic Contours from Inverse Detectors”, International Conference on Computer Vision, Barcelona Spain, Nov. 6-13, 2011, 8 pages. [cited by applicant]
Hassibi et al., “Second Order Derivatives for Network Pruning: Optimal Brain Surgeon”, Advances in Neural Information Processing Systems, Denver, Colorado, Nov. 30-Dec. 3, 1992, pp. 164-171. [cited by applicant]
He et al., Deep Residual Learning for Image Recognition, arXiv: 1512.03385v1, Dec. 10, 2015, 12 pages. [cited by applicant]
Holschneider et al., “A Real-Time Algorithm for Signal Analysis with the Help of the Wavelet Transform”, Wavelets Time-Frequency Methods and Phase Space, 1989, pp. 286-297. [cited by applicant]
Howard et al., “Efficient Convolutional Neural Networks for Mobile Vision Applications”, arXiv: 1704.04861v1, Apr. 17, 2017, 9 pages. [cited by applicant]
Huang et al., “Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors”, Conference on Computer Vision and Pattern Recognition, Honolulu, Hawaii, Jul. 21-26, 2017, 10 pages. [cited by applicant]
Huang et al., “Tensorflow Object Detection API”, https://github.com/tensorflow/models/tree/master/research/object_detection, retrieved on Feb. 13, 2018, 5 pages. [cited by applicant]
Jégou wr L., “The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation.” Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2017, 9 pages. [cited by applicant]
Jia et al., “Caffe: Convolutional Architecture for Fast Feature Embedding”, arXiv: 1408.5093v1, Jun. 20, 2014, 4 pages. [cited by applicant]
Krizhevsky et al., “ImageNet Classification with Deep Convolutional Neural Networks”, Proceedings of Neural Information Processing Systems—vol. 1, Lake Tahoe, Nevada, Dec. 3-6, 2012, pp. 1106-1114. [cited by applicant]
LeCun et al., “Optimal Brain Damage”, Advances in Neural Information Processing Systems 2, Denver, Colorado, Nov. 27-30, 1989, pp. 598-605. [cited by applicant]
Li et al., “Pruning Filters for Efficient Convnets”, arXiv:1608.08710v2, Sep. 15, 2016, 9 pages. [cited by applicant]
Lin et al., “Microsoft COCO: Common Objects in Context”, arXiv: 1405.0312v2, Jul. 5, 2014, 14 pages. [cited by applicant]
Liu et al., “ParseNet: Looking Wider to See Better”, arXiv: 1506.04579v2, Nov. 19, 2015, 11 pages. [cited by applicant]
Liu et al., “SSD: Single Shot Multibox Detector”, 14 [cited by applicant]
Papandreou et al., “Modeling Local and Global Deformations in Deep Learning: Epitomic Convolution, Multiple Instance Learning, and Sliding Window Detection”, IEEE Conference on Computer Vision and Pattern Recognition, B… [cited by applicant]
Qassim et al., “The Compressed Model of Residual CNDS”, Jun. 15, 2017, https://arxiv.org/ftp/arxiv/papers/1706/1706.06419.pdf, retrieved on Nov. 8, 2018. [cited by applicant]
Redmon et al., “Yolo9000: Better, Faster, Stronger”, arXiv: 1612.08242v1, Dec. 25, 2016, 9 pages. [cited by applicant]
Ren et al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, Advances in Neural Information Processing Systems, Montreal, Canada, Dec. 7-12, 2015, pp. 91-99. [cited by applicant]
Russakovsky et al., “ImageNet Large Scale Visual Recognition Challenge”, International Journal of Computer Vision, vol. 115, Issue 3, Dec. 2015, pp. 211-252. [cited by applicant]
Sandler, et al., “MobileNetV2: Inverted Residuals and Linear Bottlenecks”, Apr. 2, 2018, https://arxiv.org/pdf/1801.04381.pdf, retrieved on Nov. 7, 2018. [cited by applicant]
Sermanet et al., “Overfeat: Integrated Recognition, Localization and Detection Using Convolutional Networks”, arXiv:1312.6229v4, Feb. 24, 2014, 16 pages. [cited by applicant]
Simonyan et al., “Very Deep Convolutional Networks for Large-Scale Image Recognition”, arXiv: 1409.1556v5, Dec. 23, 2014, 13 pages. [cited by applicant]
Snoek et al., “Practical Bayesian Optimization of Machine Learning Algorithms”, Conference on Neural Information Processing Systems—vol. 2, Lake Tahoe, Nevada, Dec. 3-6, 2012, pp. 2960-2968. [cited by applicant]
Snoek et al., “Scalable Bayesian Optimization Using Deep Neural Networks”, Conference on Machine Learning, Lille, France, Jul. 6-11, 2015, vol. 37, pp. 2171-2180. [cited by applicant]
Szegedy et al., “Going Deeper with Convolutions”, arXiv:1409.4842v1, Sep. 17, 2014, 12 pages. [cited by applicant]
Szegedy et al., “Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning”, arXiv: 1602.07261v2, Aug. 23, 2016, 12 pages. [cited by applicant]
Veit et al., “Residual Networks are Exponential Ensembles of Relatively Shallow Networks”, arXiv: 1605.06431v2, Oct. 27, 2016, 9 pages. [cited by applicant]
Veniat et al., “Learning Time-Efficient Deep Architectures with Budgeted Super Networks”, arXiv: 1706.00046v2, Nov. 27, 2017, 13 pages. [cited by applicant]
Wang et al., “Design of Efficient Convolutional Layers Using Single Intra-Channel Convolution, Topological Subdivisioning and Spatial ‘Bottleneck’ Structure”, arXiv: 1608.04337v2, Jan. 24, 2017, 9 pages. [cited by applicant]
Written Opinion of the International Preliminary Examining Authority for Application No. PCT/US2018/046022, mailed on Nov. 5, 2019, 8 pages. [cited by applicant]
Xie et al., “Aggregated Residual Transformations for Deep Neural Networks”, arXiv: 1611.05431v2, Apr. 11, 2017, 10 pages. [cited by applicant]
Xie et al., “Genetic CNN”, arXiv: 1703.01513vl, Mar. 4, 2017, 10 pages. [cited by applicant]
Zhang et al., “ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices”, arXiv.org, Cornell University Library, Ithaca, New York, Jul. 4, 2017. [cited by applicant]
Zoph et al., “Learning Transferable Architectures for Scalable Image Recognition”, arXiv: 1707.07012v3, Dec. 1, 2017, 14 pages. [cited by applicant]
Zoph et al., “Neural Architecture Search with Reinforcement Learning”, arXiv: 1611.01578v2, Feb. 15, 2017, 16 pages. [cited by applicant]
Chinese Search Report Corresponding to Application No. 2018800669144 on Nov. 22, 2023. [cited by applicant]