System for universal hardware-neural network architecture search (co-design)
An architecture search system evaluates a search space of neural network and hardware architectures with a plurality of candidate controllers. Each controller attempts to identify an optimized architecture using a different optimization algorithm. To identify a controller for the search space, the architecture search system samples subspaces of the search space having a portion of the neural network search space and a portion of the hardware search space. For each subspace, candidate controllers are scored with respect to the optimized design determined by the respective candidate controllers. Using the scores for the various candidate controllers across the sampled subspaces, a controller is selected to optimize the overall network architecture search space.
1 . A method for optimizing a hardware-network architecture search for automated computer models, comprising:
sampling a plurality of search subspaces from a search space of neural network architectures and hardware architectures, each search subspace including a portion of the search space;
determining controller-optimized designs for the plurality of search subspaces using a plurality of candidate controllers, wherein the plurality of candidate controllers identify the controller-optimized designs using different optimization algorithms, wherein a controller-optimized design is a hardware-neural network design;
scoring the plurality of candidate controllers by determining a plurality of scores for a single candidate controller, wherein determining the plurality of scores for the single candidate controller comprises:
determining a first score for the single candidate controller based on one or more controller-optimized designs for a first search subspace and an optimal subspace design for the first search subspace, and
determining a second score for the single candidate controller based on one or more controller-optimized designs for a second search subspace and an optimal subspace design for the second search subspace;
selecting an optimization controller from the plurality of candidate controllers based on the scoring;
determining a neural network architecture and hardware architecture by applying the selected optimization controller on at least part of the search space;
generating an automated computer model with the neural network architecture; and
executing the automated computer model using a hardware device with the hardware architecture.
2 . The method of claim 1 , further comprising applying a search definition for the search space to the selected optimization controller, wherein the search definition specifies an objective of a task to be performed by the automated computer model.
3 . The method of claim 1 , wherein for each search subspace, the scoring includes all of the plurality of candidate controllers.
4 . The method of claim 1 , wherein, for each search subspace, the plurality of candidate controllers are scored in an exploration order, wherein for a particular candidate controller in the exploration order, a controller-optimized design for the particular candidate controller is evaluated with respect to a quality threshold, such that when the quality threshold is met, a subsequent controller in the exploration order is not evaluated for the search subspace, and when the quality threshold is not met, the next controller in the exploration order is evaluated.
5 . The method of claim 1 , wherein the optimal subspace design is determined from a plurality of subspace configurations by enumerating all subspace configurations within the search subspace.
6 . The method of claim 1 , wherein one or more of the plurality of candidate controllers include interfacing between a hardware controller and a neural network controller.
7 . The method of claim 6 , wherein the hardware controller or neural network controller includes differentiable algorithms, genetic algorithms, or reinforcement algorithms.
8 . The method of claim 1 , wherein the neural network architecture and hardware architecture include one or more of discrete designs, probabilistic designs, and a design population.
9 . The method of claim 1 , wherein the scoring is based on an objective included in the search definition.
10 . A system comprising:
a processor; and
a non-transitory computer-readable storage medium containing computer program code for execution by the processor for:
sampling a plurality of search subspaces from a search space of neural network architectures and hardware architectures, each search subspace including a portion of the search space,
determining controller-optimized designs for the plurality of search subspaces using a plurality of candidate controllers, wherein the plurality of candidate controllers identify the controller-optimized designs using different optimization algorithms, wherein a controller-optimized design is a hardware-neural network design,
scoring the plurality of candidate controllers by determining a plurality of scores for a single candidate controller, wherein determining the plurality of scores for the single candidate controller comprises:
determining a first score for the single candidate controller based on one or more controller-optimized designs for a first search subspace with and an optimal subspace design for the first search subspace, and
determining a second score for the single candidate controller based on one or more controller-optimized designs for a second search subspace and an optimal subspace design for the second search subspace,
selecting an optimization controller from the plurality of candidate controllers based on the scoring,
determining a neural network architecture and hardware architecture by applying the selected optimization controller on at least part of the search space,
generating an automated computer model with the neural network architecture, and
executing the automated computer model using a hardware device with the hardware architecture.
11 . The system of claim 10 , wherein the computer program code is for further execution by the processor for applying a search definition for the search space to the selected optimization controller, wherein the search definition specifies an objective of a task to be performed by the automated computer model.
12 . The system of claim 10 , wherein for each search subspace, the scoring includes all of the plurality of candidate controllers.
13 . The system of claim 10 , wherein, for each search subspace, the plurality of candidate controllers are scored in an exploration order, wherein for a particular candidate controller in the exploration order, a controller-optimized design for the particular candidate controller is evaluated with respect to a quality threshold, such that when the quality threshold is met, a subsequent controller in the exploration order is not evaluated for the search subspace, and when the quality threshold is not met, the next controller in the exploration order is evaluated.
14 . The system of claim 10 , wherein the optimal subspace design is determined from a plurality of subspace configurations by enumerating all subspace configurations within the search subspace.
15 . The system of claim 10 , wherein one or more of the plurality of candidate controllers include interfacing between a hardware controller and a neural network controller.
16 . The system of claim 15 , wherein the hardware controller or neural network controller includes differentiable algorithms, genetic algorithms, or reinforcement algorithms.
17 . The system of claim 10 , wherein the neural network architecture and hardware architecture include one or more of discrete designs, probabilistic designs, and a design population.
18 . The system of claim 10 , wherein the scoring is based on an objective included in the search definition.
19 . A non-transitory computer-readable storage medium containing instructions executable by a processor for:
sampling a plurality of search subspaces from a search space of neural network architectures and hardware architectures, each search subspace including a portion of the search space;
determining controller-optimized designs for the plurality of search subspaces using a plurality of candidate controllers, wherein the plurality of candidate controllers identify the controller-optimized designs using different optimization algorithms, wherein a controller-optimized design is a hardware-neural network design;
scoring the plurality of candidate controllers by determining a plurality of scores for a single candidate controller, wherein determining the plurality of scores for the single candidate controller comprises:
determining a first score for the single candidate controller based on one or more controller-optimized designs for a first search subspace and an optimal subspace design for the first search subspace, and
determining a second score for the single candidate controller based on one or more controller-optimized designs for a second search subspace and an optimal subspace design for the second search subspace;
selecting an optimization controller from the plurality of candidate controllers based on the scoring;
determining a neural network architecture and hardware architecture by applying the selected optimization controller on at least part of the search space;
generating an automated computer model with the neural network architecture; and
executing the automated computer model using a hardware device with the hardware architecture.
20 . The non-transitory computer-readable storage medium of claim 19 , wherein the instructions are further executable for applying a search definition for the search space to the selected optimization controller, wherein the search definition specifies an objective of a task to be performed by the automated computer model.
21 . The non-transitory computer-readable storage medium of claim 19 , wherein for each search subspace, the scoring includes all of the plurality of candidate controllers.
22 . The non-transitory computer-readable storage medium of claim 19 , wherein, for each search subspace, the plurality of candidate controllers are scored in an exploration order, wherein for a particular candidate controller in the exploration order, a controller-optimized design for the particular candidate controller is evaluated with respect to a quality threshold, such that when the quality threshold is met, a subsequent controller in the exploration order is not evaluated for the search subspace, and when the quality threshold is not met, the next controller in the exploration order is evaluated.
23 . The non-transitory computer-readable storage medium of claim 19 , wherein the optimal subspace design is determined from a plurality of subspace configurations by enumerating all subspace configurations within the search subspace.
24 . The non-transitory computer-readable storage medium of claim 19 , wherein one or more of the plurality of candidate controllers include interfacing between a hardware controller and a neural network controller.
25 . The non-transitory computer-readable storage medium of claim 24 , wherein the hardware controller or neural network controller includes differentiable algorithms, genetic algorithms, or reinforcement algorithms.