IP Library › Granted Patent US 12,361,305
Granted Patent B2
US 12,361,305 · App. 18/137,398 · Granted Jul 15, 2025

Neural architecture search for convolutional neural networks

Inventors: Vijay Vasudevan (Los Altos Hills, CA); Barret Zoph (Sunnyvale, CA); Jonathon Shlens (San Francisco, CA); Quoc V. Le (Sunnyvale, CA)
Assignee: Google LLC
G06N5/046G06N3/044G06N3/045G06N3/08G06T7/0002G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,361,305
App. No.
18/137,398
Granted
Jul 15, 2025
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures. One of the methods includes generating, using a controller neural network having controller parameters and in accordance with current values of the controller parameters, a batch of output sequences. The method includes, for each output sequence in the batch: generating an instance of a child convolutional neural network (CNN) that includes multiple instances of a first convolutional cell having an architecture defined by the output sequence; training the instance of the child CNN to perform an image processing task; and evaluating a performance of the trained instance of the child CNN on the task to determine a performance metric for the trained instance of the child CNN; and using the performance metrics for the trained instances of the child CNN to adjust current values of the controller parameters of the controller neural network.

Claims (43)

1. A neural network system comprising one or more computers and one or more non-transitory computer storage media encoded with instructions that, when executed by the one or more computers, cause the one or more computers to implement:

a sequence of convolutional cells, the sequence of convolutional cells comprising a plurality of convolutional cells of a first type configured to receive a cell input and to generate a cell output,

wherein each convolutional cell of the first type comprises a sequence of a predetermined number of operation blocks that each receive one or more respective input hidden states and generate a respective output hidden state, wherein one or more operation blocks in the sequence are configured to perform one or more convolution operations on one or more respective input hidden states that the one or more operation blocks receive, and

wherein each of the one or more operation blocks in the sequence of the predetermined number of operations blocks included in the convolutional cell of the first type is configured to:

apply a first operation to a first input hidden state to generate a first output;

apply a second operation to a second input hidden state to generate a second output; and

apply a combining operation to the first and second outputs to generate a respective output hidden state for the operation block,

wherein for each convolutional cell of the first type following the first convolutional cell of the first type in the sequence, at least one of the first input hidden state or the second input hidden state is an output of a previous convolutional cell in the sequence or an output of a previous operation block in the convolutional cell of the first type.

2. The neural network system of claim 1 , wherein each convolutional cell of the first type has a final architecture that has been generated using a computer-implemented method comprising:

generating, using a controller neural network having a plurality of controller parameters and in accordance with current values of the controller parameters, a batch of output sequences,

each output sequence in the batch defining an architecture for the convolutional cell of the first type configured to receive a cell input and to generate a cell output, and

for each output sequence in the batch:

generating an instance of a child convolutional neural network that includes multiple instances of the convolutional cell of the first type having the architecture defined by the output sequence;

training the instance of the child convolutional neural network to perform an image processing task; and

evaluating a performance of the trained instance of the child convolutional neural network on the image processing task to determine a performance metric for the trained instance of the child convolutional neural network;

using the performance metrics for the trained instances of the child convolutional neural network to adjust the current values of the controller parameters of the controller neural network; and

generating the final architecture for the convolutional cell of the first type using the controller neural network in accordance with the adjusted values of the controller parameters.

3. The neural network system of claim 2 , wherein each output sequence in the batch defines, for each of the one or more operation blocks:

a source for the first input hidden state for the operation block selected from one or more of: (i) outputs generated by one or more other components of the child convolutional neural network, (ii) an input image, or (iii) output hidden states of preceding operation blocks in the sequence of operation blocks within the convolutional cell of the first type;

a source for the second input hidden state for the operation block selected from one or more of: (i) outputs generated by one or more preceding convolutional cells in the sequence of convolutional cells, (ii) the input image, or (iii) output hidden states of preceding operation blocks in the sequence of operation blocks within the convolutional cell;

an operation type for the first operation selected from a predetermined set of convolutional neural network operations; and

an operation type for the second operation selected from the predetermined set of convolutional neural network operations.

4. The neural network system of claim 3 , wherein each output sequence in the batch further defines, for each of the one or more operation blocks of the convolutional cell of the first type:

an operation type for the combining operation selected from a predetermined set of combining operations.

5. The neural network system of claim 3 , wherein each output sequence comprises a value for a respective hyperparameter of the convolutional cell of the first type at each of a plurality of time steps.

6. The neural network system of claim 5 , wherein the controller neural network is a recurrent neural network that comprises:

one or more recurrent neural network layers that are configured to, for a given output sequence and at each time step:

receive as input the value of the respective hyperparameter at the preceding time step in the given output sequence, and to process the input to update a current hidden state of the recurrent neural network; and

a respective output layer for each time step, wherein each output layer is configured to, for the given output sequence:

receive an output layer input comprising the updated hidden state at the time step and to generate an output for the time step that defines a score distribution over possible values of the hyperparameter at the time step.

7. The neural network system of claim 5 , wherein generating, using a controller neural network having a plurality of controller parameters and in accordance with current values of the controller parameters, a batch of output sequences, comprises, for each output sequence in the batch and for each of the plurality of time steps:

providing as input to the controller neural network the value of the hyperparameter at the preceding time step in the output sequence to generate an output for the time step that defines a score distribution over possible values of the hyperparameter at the time step; and

sampling from the possible values in accordance with the score distribution to determine the value of the hyperparameter at the time step in the output sequence.

8. The neural network system of claim 2 , wherein the sequence of convolutional cells further comprises one or more convolutional cells of a second type, each of the one or more convolutional cells of the second type being configured to receive a second cell input and to generate a second cell output having a smaller height, a smaller width, or both from the second cell input, and wherein the instance of a child convolutional neural network for each output sequence also includes multiple instances of the second convolutional cell having the architecture defined by the output sequence.

9. The neural network system of claim 2 , wherein using the performance metrics for the trained instances of the child convolutional neural network to adjust the current values of the controller parameters of the controller neural network comprises:

training the controller neural network to generate output sequences that result in child convolutional neural networks having increased performance metrics using a machine learning training technique.

10. The neural network system of claim 9 , wherein the training technique is a policy gradient technique.

11. The neural network system of claim 9 , wherein the training technique is a REINFORCE technique.

12. The neural network system of claim 9 , wherein the training technique is Proximal Policy Optimization (PPO) technique.

13. The neural network system of claim 2 , wherein a number of filters of convolutional operations within the instances of the convolutional cell of the first type differs based on a position of the instances within the child convolutional neural network.

14. The neural network system of claim 2 , wherein the cell output of the convolutional cell of the first type has a same height and width as the cell input.

15. The neural network system of claim 2 , wherein training each instance of the child convolutional neural network comprises training each instance until a particular amount of time has elapsed.

16. The neural network system of claim 1 , wherein each convolutional cell of the first type is configured to: generate the cell output by combining output hidden states of blocks in the convolutional cell of the first type that are not selected as inputs to any of the blocks in the first convolutional cell of the first type.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 17, 2023
From: VASUDEVAN, VIJAY; ZOPH, BARRET; SHLENS, JONATHON; LE, QUOC V.
To: GOOGLE INC
Reel/Frame 065254/0988 →
CERTIFICATE OF CONVERSION Recorded Oct 17, 2023
From: GOOGLE INC
To: GOOGLE LLC
Reel/Frame 065255/0115 →
Continuity (4)
Continuation 16674801 · Nov 5, 2019
Continuation 16040067 · Jul 19, 2018
Provisional Application 62535806 · Jul 21, 2017
Related Publication 20230252327A1 · Aug 10, 2023
References Cited (93)
US 20160224903A1 · Talathi · 2016 [cited by applicant]
US 20170200076A1 · Vinyals et al. · 2017 [cited by applicant]
US 20170213156A1 · Hammon · 2017 [cited by applicant]
US 20180336453A1 · Merity · 2018 [cited by applicant]
CN 105637540 · 2016 [cited by applicant]
CN 105868829 · 2016 [cited by applicant]
CN 108475345 · 2018 [cited by applicant]
JP H0554012A · 1993 [cited by applicant]
Suganuma, Masanori, Shinichi Shirakawa, and Tomoharu Nagao. “A genetic programming approach to designing convolutional neural network architectures.” Proceedings of the genetic and evolutionary computation conference. 2… [cited by examiner]
Shankar, Sukrit, et al. “Refining architectures of deep convolutional neural networks.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. (Year: 2016). [cited by examiner]
Andrychowicz et al. “Learning to learn by gradient descent by gradient descent,” NIPS, Dec. 2016, 9 pages. [cited by applicant]
Angeline et al. An evolutionary Algorithm that constructs recurrent neural networks,: IEEE Transactions on Neural Networks 5.1 (1994): 54-65 (Year: 1994). [cited by applicant]
Ba et al. “Layer normalization,” arXiv 1607.06450, Jul. 21, 2016, 14 pages. [cited by applicant]
Baker et al. “Designing neural network architectures using reinforcement learning,” arXiv 1611.02167v3, Mar. 22, 2017, 18 pages. [cited by applicant]
Bergstra et al. “Algorithms for hyper-parameter optimization,” NIPS, Dec. 2011, 9 pages. [cited by applicant]
Bergstra et al. “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” International Conference on Machine Learning, Jun. 2013, 9 pages. [cited by applicant]
Bergstra et al. “Random search for hyper-parameter optimization,” Journal of Machine learning Research, Feb. 2012, 25 pages. [cited by applicant]
Chen et al. “Dual path networks,” NIPS Dec. 2017, 9 pages. [cited by applicant]
Chen et al. “Revisiting distributed synchronous sgd,” arXiv 1604.00981 Mar. 21, 2017, 10 pages. [cited by applicant]
Chollet. “Xception: Deep learning with depthwise separable convolutions,” arXiv 1610.02357v3, Apr. 4, 2017, 8 pages. [cited by applicant]
Clevert et al. “Fast and Accurate deep network learning by exponential linear units,” arXiv 1511.07289v5, Feb. 22, 2016, 14 pages. [cited by applicant]
Deng et al. “ImageNet: A large-scale hierarchical image database,” IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2009, 8 pages. [cited by applicant]
Devries et al. “Improved regularization of convolutional neural networks with cutout,” arXiv 1708.04552v2, Nov. 29, 2017, 8 pages. [cited by applicant]
Donahue et al. “Decaf: A deep convolutional activation feature for generic visual recognition,” ICML, vol. 32, Jan. 2014, 9 pages. [cited by applicant]
Duan et al. “RL2: Fast reinforcement learning via slow reinforcement learning,” arXiv 1611.02779v2, Nov. 10, 2016, 14 pages. [cited by applicant]
EP Office Action in European Application No. 18749981.9 dated Oct. 1, 2019, 3 pages. [cited by applicant]
EP Office Action in European Application No. 18749981.9, dated Mar. 11, 2021, 7 pages. [cited by applicant]
Finn et al. “Model-agnostic meta-learning for fast adaption of deep networks,” arxiv 1703.03400, Jul. 18, 2017, 13 pages. [cited by applicant]
Floreano et al. “Neuroevolution: from Architectures to learning,” Evolutionary Intelligence, Mar. 2008, 16 pages. [cited by applicant]
Fukushima. “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetic 36s, 1980, 10 pages. [cited by applicant]
Gastaldi. “Shake-shake regularization of 3-brach residual networks,” ICLR workshop, Apr. 2017, 5 pages. [cited by applicant]
Ha et al. “Hypernetworks,” International Conference on Learning Representations, Apr. 2017, 18 pages. [cited by applicant]
He et al. “Deep residual learning for image recognition,” IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, 9 pages. [cited by applicant]
He et al. “Identity mappings in deep residual networks,” arXiv 1603.05027v3, Jul. 25, 2016, 15 pages. [cited by applicant]
Hochreiter et al. “Long short-term memory,”. [cited by applicant]
Hochreiter et al. “Learning to learn using gradient descent,” Artificial Neural Networks, Aug. 2001, 8 pages. [cited by applicant]
Howard et al. “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv 1704.04861, Apr. 17, 2017, 9 pages. [cited by applicant]
Hu et al. “Squeeze-and-excitation networks, ” arXiv 1709.01507v3, Oct. 25, 2018, 14 pages. [cited by applicant]
Huang et al. “Deep networks with stochastic depth,” arXiv 1603.09382v3, Jul. 28, 2016, 16 pages. [cited by applicant]
Huang et al. “Densely connected convolutional networks,” arXiv 1608.06993v5, Jan. 28, 2018, 9 pages. [cited by applicant]
Huang et al. “Speed/accuracy trade-offs for modern convolutional object detectors,” arXiv 1611.10012v3, Apr. 25, 2017, 21 pages. [cited by applicant]
IN Office Action in Indian Application No. 20192703601, dated Nov. 10, 2021, 8 pages (with English translation). [cited by applicant]
Ioffe et al. “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” ICML Jun. 2015, 9 pages. [cited by applicant]
Jozefowicz et al. “An empirical exploration of recurrent network architectures,” ICML Jun. 2015, 9 pages. [cited by applicant]
JP Office Action in Japanese Application No. 2019-553406, dated Jan. 12, 2021, 5 pages (with English translation). [cited by applicant]
Krizhevsky et al. “Imagenet classification with deep convolutional neural networks,” NIPS Dec. 2012, 9 pages. [cited by applicant]
Krizhevsky et al. “Learning multiple layers of features from tiny images,” Technical report, University of Toronto, Apr. 8, 2009, 60 pages. [cited by applicant]
Larsson et al. “Fractalnet: Ultra-deep neural networks without residuals,” arXiv 1605.07648v4, May 26, 2017, 11 pages. [cited by applicant]
LeCun et al. “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, Nov. 1998, 46 pages. [cited by applicant]
Li et al. “Learning to optimize neural nets,” arXiv 1703.00441v2, Nov. 30, 2017, 10 pages. [cited by applicant]
Lin et al. “Focal loss for dense object detection,” arXiv 1708.02002v2, Feb. 7, 2018, 10 pages. [cited by applicant]
Lin et al. “Microsoft coco: Common objects in context,” arXiv 1405.0312v3, Feb. 21, 2015, 15 pages. [cited by applicant]
Loshchilov et al. “SGDR: Stochastic gradient descent with warm restarts,” arXiv 1608.03983v5, May 3, 2017, 16 pages. [cited by applicant]
Mendoza et al. “Towards automatically-tuned neural networks,” Proceeding of the 2016 workshop on automatic machine learning, Dec. 2016, 8 pages. [cited by applicant]
Miconi et al. “Neural networks with differentiable structure,” arXiv 1606.06216v3, Aug. 6, 2016, 4 pages. [cited by applicant]
Miikkulainen et al. “Evolving deep neural networks,” arXiv 1703.00548v2, Mar. 1, 2017, 8 pages. [cited by applicant]
Negrinho et al. “DeepArchitect: Automatically designing and training deep architectures,” arXiv 1704.08792 Apr. 28, 2017. [cited by applicant]
Neural computation, Nov. 1997, 32 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201880022762.8, dated Nov. 28, 2022, 8 pages (with English Translation). [cited by applicant]
PCT International Preliminary Report on Patentability in International Appln. No. PCT/US2018/042456, dated Jan. 21, 2020, 15 pages. [cited by applicant]
PCT International Search Report and Written Opinion issued in International Application No. PCT/US2018/042456, mailed on Nov. 12, 2018, 21 pages. [cited by applicant]
Pinto et al. “A high-throughput screening approach to discovering good forms of biologically inspired visual representation,” PLoS Computational Biology, 5(11), Nov. 2009, 12 pages. [cited by applicant]
Ravi et al. “Optimization as a model for few-shot learning,” International Conference on Learning Representations, Apr. 2017, 11 pages. [cited by applicant]
Real et al. “Large-scale evolution if image classifiers,” arXiv 1703.01041v2, Jun. 11, 2017, 18 pages. [cited by applicant]
Ren et al. “Faster R-CNN: Towards real-time object detection with region proposal networks,” Advances in Neural Information Processing Systems, Dec. 2015, 9 pages. [cited by applicant]
Saxena et al. “Convolutional neural fabrics,” NIPS, Dec. 2016 9 pages. [cited by applicant]
scholarpedia.org [online] “Metalearning,” Schaul et al., Jun. 24, 2010, [retrieved on Aug. 1, 2018] Retrieved from Internet: URL<http://scholarpedia.org/article/Metalearning> 12 pages. [cited by applicant]
Schroff et al. “Facenet: A unified embedding for face recognition and clustering,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2015, 9 pages. [cited by applicant]
Schulman et al. “Trust region policy optimization,” International Conference on Machine learning, Jun. 2015, 9 pages. [cited by applicant]
Schulman et al., “Proximal policy optimization algorithms,” arXiv, Aug. 28, 2017, 12 pages. [cited by applicant]
Shrivastava et al. “Beyond skip connections: Top-down modulation for object detection,” arXiv 1612.06851v2, Sep. 19, 2017, 11 pages. [cited by applicant]
Simonyan et al. “Very deep convolutional networks for large-scale image recognition,” arXiv 1409.1556v6, Apr. 10, 2015, 14 pages. [cited by applicant]
Snoek et al. “Practical Bayesian optimization of machine learning algorithms,” NIPS Dec. 2012, 9 pages. [cited by applicant]
Snoek et al. “Scalable Bayesian optimization using deep neural networks,” International Conference on Machine Learning, Jun. 2015, 10 pages. [cited by applicant]
Srivastava et al. “Dropout: a simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, 15(1) Jan. 2014, 30 pages. [cited by applicant]
Stanley et al. “A hypercube-based encoding for evolving large-scale neural networks,” Artificial Life, Apr. 2009, 39 pages. [cited by applicant]
Szegedy et al. “Going deeper with convolutions,” IEEE Conference on Computer Vision and Patter Recognition, Jun. 2015, 9 pages. [cited by applicant]
Szegedy et al. “Inception-v4, inception-resnet and the impact of residual convention on learning,” arXiv 1602.07261v2, Aug. 23, 2016, 12 pages. [cited by applicant]
Szegedy et al. “Rethinking the inception architecture for computer vision,” IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, 9 pages. [cited by applicant]
Ulyanov et al. “Instance normalization: The missing ingredient for fast stylization,” arXiv 1607.08022v3, Nov. 6, 2017, 6 pages. [cited by applicant]
Wang et al. “Learning to reinforcement learn,” arXiv 1611.05763v3, Jan. 23, 2017, 17 pages. [cited by applicant]
Weyand et al. “Planet-photo geolocation with convolutional neural networks,” arXiv 1602.05314, Feb. 17, 2016, 10 pages. [cited by applicant]
Wichrowska et al. “Learned optimizers that scale and generalize,” arXiv 1703.04813v4, Sep. 7, 2017, 11 pages. [cited by applicant]
Wierstra et al. “Modeling systems with internal state using evolino,” The Genetic and Evolutionary Computation Conference, Jun. 2005, 8 pages. [cited by applicant]
Williams. “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine Learning, May 1992, 28 pages. [cited by applicant]
Xie et a; “Aggregated residual transformations for deep neural networks,” arXiv 1611.05431v2, Apr. 11, 2017, 10 pages. [cited by applicant]
Xie et al. “Genetic CNN,” arXiv 1703.01513, Mar. 2017, 10 pages. [cited by applicant]
Zhang et al. “Polynet: A pursuit of structural diversity in very deep netowks,” arXiv 1611.05725v2 Jul. 17, 2017, 9 pages. [cited by applicant]
Zhang et al. “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” arXiv 1707.01083v2 Dec. 7, 2017, 9 pages. [cited by applicant]
Zhong et al. “Practical Block-wise Neural Network Architecture Generation,” arXiv 1708.05552 May 14, 2018, 11 pages. [cited by applicant]
Zoph et al. “Neural architecture search with reinforcement learning,” arXiv1611.01578v2, Feb. 15, 2017, 16 pages. [cited by applicant]
Zoph et al. “Neural Architecture Search with Reinforcement Learning,” https://arxiv.org/abs/1611.01578v1, Nov. 2016, 15 pages. [cited by applicant]
Notice of Allowance in Chinese Appln. No. 201880022762.8, mailed on Nov. 21, 2023, 6 pages (with English translation). [cited by applicant]