Neural architecture search with factorized hierarchical search space
The present disclosure is directed to an automated neural architecture search approach for designing new neural network architectures such as, for example, resource-constrained mobile CNN models. In particular, the present disclosure provides systems and methods to perform neural architecture search using a novel factorized hierarchical search space that permits layer diversity throughout the network, thereby striking the right balance between flexibility and search space size. The resulting neural architectures are able to be run relatively faster and using relatively fewer computing resources (e.g., less processing power, less memory usage, less power consumption, etc.), all while remaining competitive with or even exceeding the performance (e.g., accuracy) of current state-of-the-art mobile-optimized models.
1. A computer-implemented method, the method comprising:
defining, by one more computing devices, an initial network structure for an artificial neural network, the initial network structure comprising a plurality of blocks;
associating, by the one or more computing devices, a plurality of sub-search spaces respectively with the plurality of blocks, wherein the sub-search space for each block has one or more searchable parameters associated therewith, wherein the one or more searchable parameters included in the sub-search space associated with at least one of the plurality of blocks comprise a number of layers included in the block; and
for each of one or more iterations:
modifying, by one or more computing devices, at least one of the searchable parameters in the sub-search space associated with at least one of the plurality of blocks to generate a new network structure for the artificial neural network.
2. The computer-implemented method of claim 1 , wherein the plurality of sub-search spaces are independent from each other such that modification of at least one of the searchable parameters in one of the sub-search spaces does not necessitate modification of the searchable parameters of any other of the sub-search spaces.
3. The computer-implemented method of claim 1 , wherein, for the at least one of the plurality of blocks, the number of layers comprise a number of identical layers and the searchable parameters for such block are uniformly applied to the number of identical layers included in such block.
4. The computer-implemented method of claim 1 , wherein the one or more searchable parameters included in the sub-search space associated with at least one of the plurality of blocks comprise an operation to be performed by each of one or more layers included in the block.
5. The computer-implemented method of claim 4 , wherein a set of available operations for the searchable parameter of the operation to be performed comprise one or more of:
a convolution;
a depthwise convolution;
an inverted bottleneck convolution; or
a group convolution.
6. The computer-implemented method of claim 1 , wherein the one or more searchable parameters included in the sub-search space associated with at least one of the plurality of blocks comprise one or more of:
a kernel size;
a skip operation to be performed; or
an output filter size.
7. The computer-implemented method of claim 1 , wherein all of the plurality of sub-search spaces share a same set of searchable parameters.
8. The computer-implemented method of claim 1 , wherein at least two of the plurality of sub-search spaces have different sets of searchable parameters associated therewith.
9. The computer-implemented method of claim 1 , further comprising, for each iteration:
measuring, by the one or more computing devices, one or more performance characteristics of the new network structure for the artificial neural network.
10. The computer-implemented method of claim 9 , further comprising, for each iteration:
determining, by the one or more computing devices, a reward to provide to a controller in a reinforcement learning scheme based at least in part on the one or more performance characteristics.
11. The computer-implemented method of claim 9 , wherein the one or more performance characteristics comprise a real-world latency associated with implementation of the new network structure on a real-world mobile device.
12. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
defining an initial network structure for an artificial neural network, the initial network structure comprising a plurality of blocks, wherein a plurality of sub-search spaces are respectively associated with the plurality of blocks, the sub-search space for each block having one or more searchable parameters associated therewith; and
for each of a plurality of iterations:
modifying at least one of the searchable parameters in the sub-search space associated with at least one of the plurality of blocks to generate a new network structure for the artificial neural network.
13. The computing system of claim 12 , wherein the plurality of sub-search spaces comprise a plurality of independent sub-search spaces such that modification of at least one of the searchable parameters in one of the sub-search spaces does not necessitate modification of the searchable parameters of any other of the sub-search spaces.
14. The computing system of claim 12 , wherein the one or more searchable parameters included in the sub-search space associated with at least one of the plurality of blocks comprise a number of layers included in the block.
15. The computing system of claim 12 , wherein the one or more searchable parameters for each block are uniformly applied to all layers included in such block.
16. One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations, the operations comprising:
defining, by one more computing devices, an initial network structure for an artificial neural network, the initial structure comprising a plurality of blocks, wherein a plurality of sub-search spaces are respectively associated with the plurality of blocks, the sub-search space for each block having a plurality of searchable parameters associated therewith, the plurality of searchable parameters for each block comprising at least a number of identical layers included in the block and an operation to be performed by each of the number of identical layers included in the block; and
for each of a plurality of iterations:
modifying, by one or more computing devices, at least one of the searchable parameters in the sub-search space associated with at least one of the plurality of blocks to generate a new network structure for the artificial neural network, wherein the number of identical layers included in at least one of the plurality of blocks comprises two or more identical layers.
17. The one or more non-transitory computer-readable media of claim 16 , wherein the plurality of sub-search spaces comprise a plurality of independent sub-search spaces such that modification of at least one of the searchable parameters in one of the sub-search spaces does not necessitate modification of the searchable parameters of any other of the sub-search spaces.
18. The one or more non-transitory computer-readable media of claim 16 , wherein a set of available operations for the searchable parameter of the operation to be performed by each of the number of identical layers comprise one or more of:
a convolution;
a depthwise convolution;
a mobile inverted bottleneck convolution; or
a group convolution.
19. The one or more non-transitory computer-readable media of claim 16 , wherein the plurality of searchable parameters included in the sub-search space associated with at least one of the plurality of blocks additionally comprise one or more of:
a kernel size;
a skip operation to be performed; or
an output filter size.
20. The one or more non-transitory computer-readable media of claim 16 , wherein the plurality searchable parameters included in the sub-search space associated with at least one of the plurality of blocks additionally comprise an input size.