Hardware-aware efficient neural network design system having differentiable neural architecture search
Computer implemented systems are described that implement a differentiable neural architecture search (DNAS) engine executing on one or more processors. The DNAS engine is configured with a stochastic super net defining a layer-wise search space having a plurality of candidate layers, each of the candidate layers specifying one or more operators for a neural network architecture. Further, the DNAS engine is configured to process training data to train weights for the operators in the stochastic super net based on a loss function representing a latency of the respective operator on a target platform, and to select a set of candidate neural network architectures from the trained stochastic super net. The DNAS engine may, for example, be configured to train the stochastic super net by traversing the layer-wise search space using gradient-based optimization of network architecture distribution.
1. A computer-implemented system comprising:
a differentiable neural architecture search (DNAS) engine executing on one or more processors, wherein the DNAS engine is configured with a stochastic super net defining a layer-wise search space having a plurality of candidate layers, each of the candidate layers specifying one or more operators for a neural network architecture,
wherein the DNAS engine is configured to train the stochastic super net by processing training data to train weights for the operators in the stochastic super net, wherein the DNAS engine is configured to train a weight for a respective operator based on a loss function representing a latency of the respective operator on a target platform, wherein the latency of the respective operator is based on an estimated runtime of the respective operator on the target platform that is independent of runtimes of other operators on the target platform, and
wherein the DNAS engine is configured to select a set of candidate neural network architectures from the trained stochastic super net.
2. The system of claim 1 , wherein the DNAS engine is configured to train the stochastic super net by traversing the layer-wise search space using gradient-based optimization of network architecture distribution.
3. The system of claim 1 ,
wherein the stochastic super net comprises each of the candidate layers having a plurality of parallel candidate blocks, and
wherein the DNAS engine is configured to train the stochastic super net by sampling each of the candidate layers to select and execute one of the candidate blocks from each of the candidate layers.
4. The system of claim 1 ,
wherein the layer-wise search space defines a set of input and output dimensions of image data for each of the candidate layers, and
wherein each of the candidate layers is associated with a corresponding image block type.
5. The system of claim 1 , wherein one or more of the candidate layers is associated with a corresponding number of output filters.
6. The system of claim 1 , wherein, to train the stochastic super net, the DNAS engine is configured to access a latency lookup table that defines the estimated runtime of the respective operator on the target platform.
7. A method comprising:
constructing a stochastic super net defining a layer-wise search space having a number of candidate layers, each of the candidate layers specifying one or more operators for a neural network architecture;
training the stochastic super net by processing training data to train weights for the operators in the stochastic super net, wherein training the stochastic super net includes training a weight for a respective operator based on a loss function associated with a latency of the respective operator on a target platform, wherein the latency of the respective operator is based on an estimated runtime of the respective operator on the target platform that is independent of runtimes of other operators on the target platform; and
selecting a set of candidate neural network architectures from the trained stochastic super net.
8. The method of claim 7 , wherein training the stochastic super net comprises traversing the layer-wise search space using a gradient-based optimization of network architecture distribution.
9. The method of claim 7 ,
wherein constructing the stochastic super net comprises constructing each of the candidate layers to have a plurality of parallel candidate blocks, and
wherein training the stochastic super net comprises sampling each of the candidate layers to select and execute, with the training data, one of the candidate blocks from each of the candidate layers.
10. The method of claim 7 ,
wherein constructing the stochastic super net comprises constructing the layer-wise search space to define a set of input and output dimensions of image data for each of the candidate layers and associating each of the candidate layers with a corresponding image block type.
11. The method of claim 7 , wherein training the stochastic super net comprises accessing a latency lookup table that defines the estimated runtime of the respective operator on the target platform.
12. A computer-readable medium comprising instructions for causing one or more programmable processors to:
construct a stochastic super net defining a layer-wise search space having a number of candidate layers, each of the candidate layers specifying one or more operators for a neural network architecture;
train the stochastic super net by processing training data to train weights for the operators in the stochastic super net, wherein the instructions cause the one or more programmable processors to train a weight for a respective operator based on a loss function associated with a latency of the respective operator on a target platform, wherein the latency of the respective operator is based on an estimated runtime of the respective operator on the target platform that is independent of runtimes of other operators on the target platform; and
select a set of candidate neural network architectures from the trained stochastic super net.
13. The computer-readable medium of claim 12 , further comprising instructions to train the stochastic super net by traversing the layer-wise search space using a gradient-based optimization of network architecture distribution.
14. The computer-readable medium of claim 12 , further comprising instructions to:
construct the stochastic super net by constructing each of the candidate layers to have a plurality of parallel candidate blocks, and
train the stochastic super net by sampling each of the candidate layers to select and execute, with the training data, one of the candidate blocks from each of the candidate layers.
15. The computer-readable medium of claim 12 , further comprising instructions to
construct the stochastic super net by constructing the layer-wise search space to define a set of input and output dimensions of image data for each of the candidate layers and associating each of the candidate layers with a corresponding image block type.
16. The computer-readable medium of claim 12 , further comprising instructions to train the stochastic super net by accessing a latency lookup table that defines the estimated runtime of the respective operator on the target platform, and computing an overall latency of one or more candidate convolutional neural network models according to estimated runtimes for the operators in the stochastic super net.