IP Library › Granted Patent US 12,632,737
Granted Patent B2
US 12,632,737 · App. 18/415,376 · Granted May 19, 2026

Using hierarchical representations for neural network architecture searching

Inventors: Chrisantha Thomas Fernando (London, GB); Karen Simonyan (London, GB); Koray Kavukcuoglu (London, GB); Hanxiao Liu (Santa Clara, CA); Oriol Vinyals (London, GB)
Assignee: GDM Holding LLC
G06N3/086G06F16/9024G06N3/045G06F17/15
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,632,737
App. No.
18/415,376
Granted
May 19, 2026
Kind
B2
Abstract

A computer-implemented method for automatically determining a neural network architecture represents a neural network architecture as a data structure defining a hierarchical set of directed acyclic graphs in multiple levels. Each graph has an input, an output, and a plurality of nodes between the input and the output. At each level, a corresponding set of the nodes are connected pairwise by directed edges which indicate operations performed on outputs of one node to generate an input to another node. Each level is associated with a corresponding set of operations. At a lowest level, the operations associated with each edge are selected from a set of primitive operations. The method includes repeatedly generating new sample neural network architectures, and evaluating their fitness. The modification is performed by selecting a level, selecting two nodes at that level, and modifying, removing or adding an edge between those nodes according to operations associated with lower levels of the hierarchy.

Claims (47)

1 . A computer-implemented method for automatically determining a neural network architecture, comprising:

generating data defining a hierarchical set of graphs comprising a succession of hierarchical levels of graphs, each graph comprising a plurality of nodes and one or more edges that each connect two respective nodes, wherein: each node represents a feature map within a neural network, each edge represents an operation performed on a feature map represented by an input node of the edge to provide a feature map represented by an output node of the edge, wherein the succession of hierarchical levels include a lowest hierarchical level and a plurality of additional hierarchical levels, and each additional hierarchical level of graphs comprises graphs that are assembled using graphs selected from one or more graphs of a preceding hierarchical level, and generating the data comprises:

generating data for one or more graphs of the lowest hierarchical level, wherein operations corresponding to edges of the one or more graphs in the lowest hierarchical level are selected from a set of primitive neural network operations, and

recursively generating data defining, for each of the plurality of additional hierarchical levels, a respective set of one or more graphs, wherein the operations corresponding to the edges of the one or more graphs in the respective additional hierarchical level are selected from a set of operations performed by one or more graphs in a hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels;

initiating a population of two or more sample neural network architectures using the hierarchical set of graphs;

generating sample neural networks having the sample neural network architectures in the population;

training the sample neural networks;

evaluating the sample neural networks by determining a fitness value for each of the sample neural networks; and

selecting one or more of the sample neural network architectures in the population according to the determined fitness values to determine a neural network architecture.

2 . The method of claim 1 , wherein the set of primitive neural network operations includes an identity operation that when performed on a feature map leaves the feature map unchanged.

3 . The method of claim 1 , wherein the set of primitive neural network operations includes at least one convolution operation that when performed on a feature map leaves a resolution of the feature map unchanged.

4 . The method of claim 1 , wherein the set of primitive neural network operations includes at least one convolution operation followed by a batch normalization operation.

5 . The method of claim 1 , wherein the set of primitive neural network operations includes a no-connection operation which defines that there is no direct connection between the nodes linked by the edge to which the operation corresponds.

6 . The method of claim 1 , wherein the set of primitive neural network operations includes an identity operation that when performed on a feature map leaves the feature map unchanged and the initializing comprises initializing at least some of the operations performed by the edges of the hierarchical set of graphs to the identity operation.

7 . The method of claim 1 , wherein the selecting comprises comparing the fitness values of sample neural network architectures from the population against one another.

8 . The method of claim 1 , wherein determining at least two sample neural network architectures further comprises mutating a sample neural network architecture selected according to the determined fitness value and repeating the generating, training, evaluating and selecting to evolve the sample neural network architectures.

9 . The method of claim 8 , wherein the mutating comprises selecting one of the hierarchical levels, selecting a graph in the selected level, selecting a predecessor node and a successor node in the selected graph, and replacing an operation corresponding to an edge connecting the selected nodes with another operation.

10 . The method of claim 1 , wherein the selecting further comprises:

providing a plurality of evaluation workers each configured to evaluate a sample neural network architecture;

allocating sample neural network architectures to the evaluation workers for evaluation as each worker becomes available, wherein each worker performs the generating, training and evaluating of a sample neural network having a sample neural network architecture and, when finished, is allocated a further sample neural network architecture;

adding results of the evaluations to a data store shared by the evaluation workers; and

controlling the selecting using the evaluations in the data store.

11 . The method of claim 1 , further comprising constructing a neural network according to the determined neural network architecture.

12 . The method of claim 11 , further comprising using the neural network in a neural network system for training and/or inference.

13 . The method of claim 11 , further comprising making the neural network available in a neural network system for training and/or inference via an API.

14 . The method according to claim 12 , wherein the neural network system includes multiple instances of the neural network.

15 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for automatically determining a neural network architecture, the operations comprising:

generating data defining a hierarchical set of graphs comprising a succession of hierarchical levels of graphs, each graph comprising a plurality of nodes and one or more edges that each connect two respective nodes, wherein: each node represents a feature map within a neural network, each edge represents an operation performed on a feature map represented by an input node of the edge to provide a feature map represented by an output node of the edge, wherein the succession of hierarchical levels include a lowest hierarchical level and a plurality of additional hierarchical levels, and each additional hierarchical level of graphs comprises graphs that are assembled using graphs selected from one or more graphs of a preceding hierarchical level, and generating the data comprises:

generating data for one or more graphs of the lowest hierarchical level, wherein operations corresponding to edges of the one or more graphs in the lowest hierarchical level are selected from a set of primitive neural network operations, and

recursively generating data defining, for each of the plurality of additional hierarchical levels, a respective set of one or more graphs, wherein the operations corresponding to the edges of the one or more graphs in the respective additional hierarchical level are selected from a set of operations performed by one or more graphs in a hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels;

initiating a population of two or more sample neural network architectures using the hierarchical set of graphs;

generating sample neural networks having the sample neural network architectures in the population;

training the sample neural networks;

evaluating the sample neural networks by determining a fitness value for each of the sample neural networks; and

selecting one or more of the sample neural network architectures in the population according to the determined fitness values to determine a neural network architecture.

16 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for automatically determining a neural network architecture, the operations comprising:

generating data defining a hierarchical set of graphs comprising a succession of hierarchical levels of graphs, each graph comprising a plurality of nodes and one or more edges that each connect two respective nodes, wherein: each node represents a feature map within a neural network, each edge represents an operation performed on a feature map represented by an input node of the edge to provide a feature map represented by an output node of the edge, wherein the succession of hierarchical levels include a lowest hierarchical level and a plurality of additional hierarchical levels, and each additional hierarchical level of graphs comprises graphs that are assembled using graphs selected from one or more graphs of a preceding hierarchical level, and generating the data comprises:

generating data for one or more graphs of the lowest hierarchical level, wherein operations corresponding to edges of the one or more graphs in the lowest hierarchical level are selected from a set of primitive neural network operations, and

recursively generating data defining, for each of the plurality of additional hierarchical levels, a respective set of one or more graphs, wherein the operations corresponding to the edges of the one or more graphs in the respective additional hierarchical level are selected from a set of operations performed by one or more graphs in a hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels;

initiating a population of two or more sample neural network architectures using the hierarchical set of graphs;

generating sample neural networks having the sample neural network architectures in the population;

training the sample neural networks;

evaluating the sample neural networks by determining a fitness value for each of the sample neural networks; and

selecting one or more of the sample neural network architectures in the population according to the determined fitness values to determine a neural network architecture.

17 . The system of claim 15 , wherein determining at least two sample neural network architectures includes initializing one or a population of sample neural network architectures by:

initializing at least one instance of the data structure defining the hierarchical set of graphs by defining at least the operations performed by the edges of the hierarchical set of graphs, and

mutating the at least one instance of the data structure by modifying the operations performed by the edges of the hierarchical set of graphs.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 6, 2025
From: DEEPMIND TECHNOLOGIES LIMITED
To: GDM HOLDING LLC
Reel/Frame 071498/0210 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 18, 2024
From: FERNANDO, CHRISANTHA THOMAS; SIMONYAN, KAREN; KAVUKCUOGLU, KORAY; LIU, HANXIAO; VINYALS, ORIOL
To: DEEPMIND TECHNOLOGIES LIMITED
Reel/Frame 066162/0535 →
Continuity (3)
Continuation 16759567
Provisional Application 62578356 · Oct 27, 2017
Related Publication 20240249146A1 · Jul 25, 2024
References Cited (44)
US 10521729B2 · Vasudevan et al. · 2019 [cited by applicant]
CN 105719001 · 2016 [cited by applicant]
CN 106471526 · 2017 [cited by applicant]
Real, Esteban, et al. “Large-scale evolution of image classifiers.” International Conference on Machine Learning. PMLR (Year: 2017). [cited by examiner]
Suganuma, Masanori, et al. “A genetic programming approach to designing convolutional neural network architectures.” Proceedings of the genetic and evolutionary computation conference. (Year: 2017). [cited by examiner]
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (Year: 2016). [cited by examiner]
Baker et al., “Designing neural network architectures using reinforcement learning,” CoRR, 2016, https://arxiv.org/abs/1611.02167, 18 pages. [cited by applicant]
Brock et al., “Smash: One-shot model architecture search through hypernetworks,” CoRR, 2017, https://arxiv.org/abs/1708.05344, 21 pages. [cited by applicant]
Chollet et al., “Xception: Deep learning with depthwise separable convolutions,” CoRR, 2016, arXivpreprintarXiv:1610.02357, 8 pages. [cited by applicant]
Desell et al., “Large scale evolution of convolutional neural networks using volunteer computing,” CoRR, 2017, arXiv:1703.05422vl, 17 pages. [cited by applicant]
Floreano et al., “Neuroevolution: from architectures to learning,” Evolutionary Intelligence, 2008, 1(1):47-62. [cited by applicant]
Goldberg et al., “A comparative analysis of selection schemes used in genetic algorithms,” Foundations of genetic algorithms, 1991, 1:69-93. [cited by applicant]
Gruau et al., “Neural network synthesis using cellular encoding and the genetic algorithm,” Thesis for the degree of Doctor of Computer Science, Universite Claude Bernard-Lyon, Jan. 4, 1994, 159 pages. [cited by applicant]
He et al., “Deep residual learning for image recognition,” Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778. [cited by applicant]
He et al., “Identity mappings in deep residual networks,” ECCV 2016: Computer Vision, 2016, pp. 630-645. [cited by applicant]
Huang et al., “Densely connected convolutional networks, ”CoRR, 2016, arXivpreprint arXiv:1608.06993, 9 pages. [cited by applicant]
Ioffe et al. “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” CoRR, 2015, https://arxiv.org/abs/1502.03167, 11 pages. [cited by applicant]
Kitano, “Designing neural networks using genetic algorithms with graph generation system,” Complex systems, 1990, 4(4):461-476. [cited by applicant]
Larsson et al., “Fractalnet: Ultra-deep neural networks without residuals,” CoRR, 2016, arXivpreprint arXiv:1605.07648, 11 pages. [cited by applicant]
Miikkulainen et al., “Evolving deep neural networks,” CoRR, 2017, arXiv preprintarXiv:1703.00548, 8 pages. [cited by applicant]
Miller et al., “Designing neural networks using genetic algorithms,” ICGA, 1989, 89:379-384. [cited by applicant]
Negrinho et al., “Deeparchitect: Automatically designing and training deep architectures,” CoRR, 2017, arXivpreprintarXiv:1704.08792, 12 pages. [cited by applicant]
Office Action in Chinese Appln. No. 201880070077.2, dated Oct. 10, 2023, 22 pages (with English translation). [cited by applicant]
PCT International Preliminary Report on Patentability in International Appln. PCT/EP2018/079401, mailed May 7, 2020, 8 pages. [cited by applicant]
PCT International Search Report and Written Opinion in International Appln. No. PCT/EP2018/079401, mailed Feb. 15, 2019, 14 pages. [cited by applicant]
Real et al., “Large-scale evolution of image classifiers,” CoRR, 2017, arxivpreprint arXiv:1703.01041, 18 pages. [cited by applicant]
Real et al., “Regularized evolution for image classifier architecture search,” CoRR, 2018, arXivpreprintarXiv:1802.01548, 10 pages. [cited by applicant]
Russakovsky et al., “Imagenet large scale visual recognition challenge. International Journal of Computer Vision,” Int J Comput Vis., 2015, 115(3):211-252. [cited by applicant]
Shafiee et al., “Learning efficient 1-19 deep feature representations via transgenerational genetic transmission of environmental information during evolutionary synthesis of deep neural networks,” 2017 IEEE Internation… [cited by applicant]
Simonyan et al., “Very deep convolutional networks for large-scale image recognition,” CoRR, 2014, arXivpreprintarXiv:1409.1556, 14 pages. [cited by applicant]
Stanley et al., “A hypercube-based encoding for evolving large-scale neural networks,” Artificial life, 2009, 15(2):185-212. [cited by applicant]
Stanley et al., “Compositional pattern producing networks: A novel abstraction of development,” Genetic programming and evolvable machines, 2007, 8(2):131-162. [cited by applicant]
Stanley et al., “Evolving neural networks through augmenting topologies,” Evolutionary computation, 2002, 10(2):99-127. [cited by applicant]
Suganuma et al., “A genetic programming approach to designing convolutional neural network architectures, ” CoRR, 2017, arXiv:1704.00764v2, 9 pages. [cited by applicant]
Szegedy et al., “Going deeper with convolutions,” CoRR, 2015, https://arxiv.org/abs/1409.4842, 12 pages. [cited by applicant]
Szegedy et al., “Inception-v4, inception-resnet and the impact of residual connections on learning,” Thirty-First AAAI Conference on Artificial Intelligence, 2017, pp. 4278-4284. [cited by applicant]
Szegedy et al., “Rethinking the inception architecture for computer vision,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818-2826. [cited by applicant]
Xie et al., “Aggregated residual transformations for deep neural networks,” CoRR, 2016, arXivpreprintarXiv:1611.05431, 10 pages. [cited by applicant]
Xie et al., “Genetic CNN,” CoRR, 2017, arXivpreprintarXiv:1703.01513, 10 pages. [cited by applicant]
Yao et al., “Evolving artificial neural networks,” Proceedings of the IEEE, 1999, 87(9):1423-1447. [cited by applicant]
Zagoruyko et al., “Wide residual networks,” CoRR, 2016, arXivpreprint arXiv:1605.07146, 15 pages. [cited by applicant]
Zhong et al., “Practical Block-wise Neural Network Architecture Generation,” CoRR, Aug. 2017, https://arxiv.org/abs/1708.05552, 11 pages. [cited by applicant]
Zoph et al., “Learning transferable architectures for scalable image recognition,” CoRR, 2017, arXivpreprintarXiv:1707.07012, 14 pages. [cited by applicant]
Zoph et al., “Neural architecture search with reinforcement learning,” CoRR, 2016, arXivpreprintarXiv:1611.01578, 16 pages. [cited by applicant]