Using hierarchical representations for neural network architecture searching
A computer-implemented method for automatically determining a neural network architecture represents a neural network architecture as a data structure defining a hierarchical set of directed acyclic graphs in multiple levels. Each graph has an input, an output, and a plurality of nodes between the input and the output. At each level, a corresponding set of the nodes are connected pairwise by directed edges which indicate operations performed on outputs of one node to generate an input to another node. Each level is associated with a corresponding set of operations. At a lowest level, the operations associated with each edge are selected from a set of primitive operations. The method includes repeatedly generating new sample neural network architectures, and evaluating their fitness. The modification is performed by selecting a level, selecting two nodes at that level, and modifying, removing or adding an edge between those nodes according to operations associated with lower levels of the hierarchy.
1. A computer-implemented method for automatically determining a neural network architecture, comprising:
generating data representing the neural network architecture as a data structure defining a hierarchical set of graphs comprising a succession of hierarchical levels of graphs, each graph having an input, an output, a plurality of nodes between the input and the output, and one or more edges that each connect two respective nodes, wherein: a node corresponds to a feature map within a neural network defined by the neural network architecture, each edge connects an input node of the edge to an output node of the edge and corresponds to an operation performed on a feature map of the input node of the edge to provide a feature map to the output node of the edge, such that each graph performs an operation, the succession of hierarchical levels include a lowest hierarchical level and a plurality of additional hierarchical levels, generating the data comprises defining the hierarchical set of graphs, and defining the hierarchical set of graphs comprises:
defining the lowest hierarchical level comprising one or more lowest level graphs, wherein operations corresponding to edges of the one or more lowest level graphs are selected from a set of primitive neural network operations, and
recursively defining the additional hierarchical levels, wherein each respective additional hierarchical level comprises a respective set of one or more graphs, wherein recursively defining each respective additional hierarchical level comprises, for each graph in the one or more graphs in the respective additional hierarchical level, selecting a set of lower-level graphs from the graphs in the hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels, and generating data representing the graph by assembling the lower-level graphs selected from the graphs in the hierarchical level that precedes the respective additional hierarchical level;
initiating a population of two or more sample neural network architectures based on the data structure defining the hierarchical set of graphs, wherein each respective sample neural network architecture in the population is initiated by modifying one or more of the operations performed by the edges of the hierarchical set of graphs;
generating sample neural networks having the sample neural network architectures in the population;
training the sample neural networks;
evaluating the sample neural networks by determining a fitness value for each of the sample neural networks; and
selecting one or more of the sample neural network architectures in the population according to the determined fitness values to determine a neural network architecture.
2. The method of claim 1 , wherein for each respective additional hierarchical level, the operations corresponding to the edges of the one or more graphs in the respective additional hierarchical level are selected from a set of operations performed by one or more graphs in a hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels.
3. The method of claim 1 , wherein the set of primitive neural network operations includes an identity operation that when performed on a feature map leaves the feature map unchanged.
4. The method of claim 1 , wherein the set of primitive neural network operations includes at least one convolution operation that when performed on a feature map leaves a resolution of the feature map unchanged.
5. The method of claim 1 , wherein the set of primitive neural network operations includes at least one convolution operation followed by a batch normalization operation.
6. The method of claim 1 , wherein the set of primitive neural network operations includes a no-connection operation which defines that there is no direct connection between the nodes linked by the edge to which the operation corresponds.
7. The method of claim 1 , wherein the set of primitive neural network operations includes an identity operation that when performed on a feature map leaves the feature map unchanged and the initializing comprises initializing at least some of the operations performed by the edges of the hierarchical set of graphs to the identity operation.
8. The method of claim 1 , wherein the selecting comprises comparing the fitness values of sample neural network architectures from the population against one another.
9. The method of claim 1 , wherein determining at least two sample neural network architectures further comprises mutating a sample neural network architecture selected according to the determined fitness value and repeating the generating, training, evaluating and selecting to evolve the sample neural network architectures.
10. The method of claim 9 , wherein the mutating comprises selecting one of the hierarchical levels, selecting a graph in the selected level, selecting a predecessor node and a successor node in the selected graph, and replacing an operation corresponding to an edge connecting the selected nodes with another operation.
11. The method of claim 1 , wherein the selecting further comprises:
providing a plurality of evaluation workers each configured to evaluate a sample neural network architecture;
allocating sample neural network architectures to the workers for evaluation as each worker becomes available, wherein each worker performs the generating, training and evaluating of a sample neural network having a sample neural network architecture and, when finished, is allocated a further sample neural network architecture;
adding results of the evaluations to a data store shared by the evaluation workers; and
controlling the selecting using the evaluations in the data store.
12. The method of claim 1 , further comprising constructing a neural network according to the determined neural network architecture.
13. The method of claim 12 , further comprising using the neural network in a neural network system for training and/or inference.
14. The method of claim 12 , further comprising making the neural network available in a neural network system for training and/or inference via an API.
15. The method according to claim 13 , wherein the neural network system includes multiple instances of the neural network.
16. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for automatically determining a neural network architecture, the operations comprising:
generating data representing the neural network architecture as a data structure defining a hierarchical set of graphs comprising a succession of hierarchical levels of graphs, each graph having an input, an output, a plurality of nodes between the input and the output, and one or more edges that each connect two respective nodes, wherein: a node corresponds to a feature map within a neural network defined by the neural network architecture, each edge connects an input node of the edge to an output node of the edge and corresponds to an operation performed on a feature map of the input node of the edge to provide a feature map to the output node of the edge, such that each graph performs an operation, the succession of hierarchical levels include a lowest hierarchical level and a plurality of additional hierarchical levels, generating the data comprises defining the hierarchical set of graphs, and defining the hierarchical set of graphs comprises:
defining the lowest hierarchical level comprising one or more lowest level graphs, wherein operations corresponding to edges of the one or more lowest level graphs are selected from a set of primitive neural network operations, and
recursively defining the additional hierarchical levels, wherein each respective additional hierarchical level comprises a respective set of one or more graphs, wherein recursively defining each respective additional hierarchical level comprises, for each graph in the one or more graphs in the respective additional hierarchical level, selecting a set of lower-level graphs from the graphs in the hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels, and generating data representing the graph by assembling the lower-level graphs selected from the graphs in the hierarchical level that precedes the respective additional hierarchical level;
initiating a population of two or more sample neural network architectures based on the data structure defining the hierarchical set of graphs, wherein each respective sample neural network architecture in the population is initiated by modifying one or more of the operations performed by the edges of the hierarchical set of graphs;
generating sample neural networks having the sample neural network architectures in the population;
training the sample neural networks;
evaluating the sample neural networks by determining a fitness value for each of the sample neural networks; and
selecting one or more of the sample neural network architectures in the population according to the determined fitness values to determine a neural network architecture.
17. The system of claim 16 , wherein for each respective additional hierarchical level, the operations corresponding to the edges of the one or more graphs in the respective additional hierarchical level are selected from a set of operations performed by one or more graphs in a hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels.
18. The system of claim 16 , wherein determining at least two sample neural network architectures includes initializing one or a population of sample neural network architectures by:
initializing at least one instance of the data structure defining the hierarchical set of graphs by defining at least the operations performed by the edges of the hierarchical set of graphs, and
mutating the at least one instance of the data structure by modifying the operations performed by the edges of the hierarchical set of graphs.
19. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for automatically determining a neural network architecture, the operations comprising:
generating data representing the neural network architecture as a data structure defining a hierarchical set of graphs comprising a succession of hierarchical levels of graphs, each graph having an input, an output, a plurality of nodes between the input and the output, and one or more edges that each connect two respective nodes, wherein: a node corresponds to a feature map within a neural network defined by the neural network architecture, each edge connects an input node of the edge to an output node of the edge and corresponds to an operation performed on a feature map of the input node of the edge to provide a feature map to the output node of the edge, such that each graph performs an operation, the succession of hierarchical levels include a lowest hierarchical level and a plurality of additional hierarchical levels, generating the data comprises defining the hierarchical set of graphs, and defining the hierarchical set of graphs comprises:
defining the lowest hierarchical level comprising one or more lowest level graphs, wherein operations corresponding to edges of the one or more lowest level graphs are selected from a set of primitive neural network operations, and
recursively defining the additional hierarchical levels, wherein each respective additional hierarchical level comprises a respective set of one or more graphs, wherein recursively defining each respective additional hierarchical level comprises, for each graph in the one or more graphs in the respective additional hierarchical level, selecting a set of lower-level graphs from the graphs in the hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels, and generating data representing the graph by assembling the lower-level graphs selected from the graphs in the hierarchical level that precedes the respective additional hierarchical level;
initiating a population of two or more sample neural network architectures based on the data structure defining the hierarchical set of graphs, wherein each respective sample neural network architecture in the population is initiated by modifying one or more of the operations performed by the edges of the hierarchical set of graphs;
generating sample neural networks having the sample neural network architectures in the population;
training the sample neural networks;
evaluating the sample neural networks by determining a fitness value for each of the sample neural networks; and
selecting one or more of the sample neural network architectures in the population according to the determined fitness values to determine a neural network architecture.