Hyper-parameter selection for deep convolutional networks
Hyper-parameters are selected for training a deep convolutional network by selecting a number of network architectures as part of a database. Each of the network architectures includes one or more local logistic regression layer and is trained to generate a corresponding validation error that is stored in the database. A threshold error for identifying a good set of network architectures and a bad set of network architectures may be estimated based on validation errors in the database. The method also includes choosing a next potential hyper-parameter, corresponding to a next network architecture, based on a metric that is a function of the good set of network architectures. The method further includes selecting a network architecture, from among next network architectures, with a lowest validation error.
1. A method of selecting hyper-parameters for training a deep convolutional network, comprising:
selecting network architectures as part of a database, each of the network architectures including at least one local logistic regression layer;
training each of the network architectures to generate a corresponding validation error that is stored in the database;
identifying, from the network architectures, a good set of network architectures and a bad set of network architectures, each network architecture of the good set of network architectures having a validation error that is less than a validation error threshold and a memory footprint that is less than a memory threshold, and each network architecture of the bad set of network architectures having a validation error that is greater than the validation error threshold and a memory footprint that is greater than the memory threshold;
choosing a potential hyper-parameter to define next network architectures, the potential hyper-parameter selected based at least in part on a metric that is a function of the good set of network architectures; and
selecting a network architecture, after training each of the next network architectures to generate a validation error, with a lowest validation error.
2. The method of claim 1 , in which each of the next network architectures includes at least one logistic regression layer.
3. The method of claim 1 , further comprising selecting learning hyper-parameters based at least in part on the selected network architecture.
4. The method of claim 1 , in which selecting the network architecture comprises jointly selecting architecture hyper-parameters and learning hyper-parameters.
5. The method of claim 1 , further comprising choosing the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from the good set of network architectures and evaluating the potential hyper-parameter based at least in part on a distribution of the good set of network architectures.
6. The method of claim 1 , further comprising choosing the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from the good set of network architectures and evaluating the potential hyper-parameter based at least in part on a ratio of a distribution of the good set of network architectures and a distribution of the bad set of network architectures.
7. The method of claim 1 , further comprising choosing the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from a random distribution and evaluating the potential hyper-parameter based at least in part on a ratio of a distribution of the good set of network architectures and a distribution of the bad set of network architectures.
8. The method of claim 1 , further comprising choosing the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from a random distribution and evaluating the potential hyper-parameter based at least in part on a distribution of the good set of network architectures.
9. The method of claim 1 , further comprising choosing the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from either the good set of network architectures or a random distribution, and evaluating the potential hyper-parameter based at least in part on either a ratio of a distribution of the good set of network architectures and a distribution of the bad set of network architectures or the distribution of the good set of network architectures based at least in part on a probability distribution.
10. An apparatus for selecting hyper-parameters for training a deep convolutional network, comprising:
a memory; and
at least one processor coupled to the memory and configured:
to select network architectures as part of a database, each of the network architectures including at least one local logistic regression layer;
to train each of the network architectures to generate a corresponding validation error that is stored in the database;
to identify, from the network architectures, a good set of network architectures and a bad set of network architectures, each network architecture of the good set of network architectures having a validation error that is less than a validation error threshold and a memory footprint that is less than a memory threshold, and each network architecture of the bad set of network architectures having a validation error that is greater than the validation error threshold and a memory footprint that is greater than the memory threshold;
to choose a potential hyper-parameter to define next network architectures, the potential hyper-parameter selected based at least in part on a metric that is a function of the good set of network architectures; and
to select a network architecture, after training each of the next network architectures to generate a validation error, with a lowest validation error.
11. The apparatus of claim 10 , in which each of the next network architectures includes at least one logistic regression layer.
12. The apparatus of claim 10 , in which the at least one processor is further configured to select learning hyper-parameters based at least in part on the selected network architecture.
13. The apparatus of claim 10 , in which the at least one processor is further configured to select the network architecture by jointly selecting architecture hyper-parameters and learning hyper-parameters.
14. The apparatus of claim 10 , in which the at least one processor is further configured to choose the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from the good set of network architectures and evaluating the potential hyper-parameter based at least in part on a distribution of the good set of network architectures.
15. The apparatus of claim 10 , in which the at least one processor is further configured to choose the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from the good set of network architectures and evaluating the potential hyper-parameter based at least in part on a ratio of a distribution of the good set of network architectures and a distribution of the bad set of network architectures.
16. The apparatus of claim 10 , in which the at least one processor is further configured to choose the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from a random distribution and evaluating the potential hyper-parameter based at least in part on a ratio of a distribution of the good set of network architectures and a distribution of the bad set of network architectures.
17. The apparatus of claim 10 , in which the at least one processor is further configured to choose the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from a random distribution and evaluating the potential hyper-parameter based at least in part on a distribution of the good set of network architectures.
18. The apparatus of claim 10 , in which the at least one processor is further configured to choose the potential hyper-parameter by selecting at least one of an architecture hyper-parameter or a learning hyper-parameter from either the good set of network architectures or a random distribution, and evaluating the potential hyper-parameter based at least in part on either a ratio of a distribution of the good set of network architectures and a distribution of the bad set of network architectures or the distribution of the good set of network architectures based at least in part on a probability distribution.
19. An apparatus for selecting hyper-parameters for training a deep convolutional network, comprising:
means for selecting network architectures as part of a database, each of the network architectures including at least one local logistic regression layer;
means for training each of the network architectures to generate a corresponding validation error that is stored in the database;
means for identifying, from the network architectures, a good set of network architectures and a bad set of network architectures, each network architecture of the good set of network architectures having a validation error that is less than a validation error threshold and a memory footprint that is less than a memory threshold, and each network architecture of the bad set of network architectures having a validation error that is greater than the validation error threshold and a memory footprint that is greater than the memory threshold;
means for choosing a potential hyper-parameter to define next network architectures, the potential hyper-parameter selected based at least in part on a metric that is a function of the good set of network architectures; and
means for selecting a network architecture, after training each of the next network architectures to generate a validation error, with a lowest validation error.
20. A non-transitory computer-readable medium having encoded thereon program code for selecting hyper-parameters for training a deep convolutional network, the program code being executed by a processor and comprising:
program code to select network architectures as part of a database, each of the network architectures including at least one local logistic regression layer;
program code to train each of the network architectures to generate a corresponding validation error that is stored in the database;
program code to identify, from the network architectures, a good set of network architectures and a bad set of network architectures, each network architecture of the good set of network architectures having a validation error that is less than a validation error threshold and a memory footprint that is less than a memory threshold, and each network architecture of the bad set of network architectures having a validation error that is greater than the validation error threshold and a memory footprint that is greater than the memory threshold;
program code to choose a potential hyper-parameter to define next network architectures, the potential hyper-parameter selected based at least in part on a metric that is a function of the good set of network architectures; and
program code to select a network architecture, after training each of the next network architectures to generate a validation error, with a lowest validation error.