Generating hyper-parameters for machine learning models using modified Bayesian optimization based on accuracy and training efficiency

1. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:

identify observed hyper-parameter sets for a machine learning architecture comprising a plurality of tunable parameters;

generate a hyper-parameter set that balances accuracy and efficiency of learning the tunable parameters by:

fitting an accuracy regression model to pairs of the observed hyper-parameter sets and associated accuracy metrics;

fitting an efficiency regression model to pairs of the observed hyper-parameter sets and associated training efficiency metrics; and

selecting hyper-parameters for the hyper-parameter set utilizing a weighted combination of the accuracy regression model and the efficiency regression model; and

learn the plurality of tunable parameters of the machine learning architecture utilizing the hyper-parameter set.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to select the hyper-parameters for the hyper-parameter set by further utilizing a trade-off acquisition function to weight an accuracy acquisition function associated with the accuracy regression model relative to an efficiency acquisition function associated with the efficiency regression model utilizing an accuracy-training efficiency balance metric.

3. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:

determine new accuracy metrics and new training efficiency metrics associated with the hyper-parameter set based on performance of the machine learning architecture.

4. The non-transitory computer readable medium of claim 3 , further comprising instructions that, when executed by the at least one processor, cause the computing device to fit a modified accuracy regression model and a modified efficiency regression model based on the new accuracy metrics and the new training efficiency metrics.

5. The non-transitory computer readable medium of claim 4 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate a new hyper-parameter set utilizing a modified trade-off acquisition function that is based on a combination of the modified accuracy regression model and the modified efficiency regression model.

6. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to fit the accuracy regression model to pairs of the observed hyper-parameter sets and associated accuracy metrics by determining probability distributions of accuracy metrics corresponding to hyper-parameter sets conditioned by the observed hyper-parameter sets and the associated accuracy metrics.

7. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to fit the efficiency regression model to pairs of the observed hyper-parameter sets and associated training efficiency metrics by determining probability distributions of training efficiency metrics corresponding to hyper-parameter sets conditioned by the observed hyper-parameter sets and the associated training efficiency metrics.

8. The non-transitory computer readable medium of claim 1 , wherein the hyper-parameter set comprises a training set size and one or more of a dropout rate, a regularization parameter, a hidden unit, a batch size, a number of epochs, a bias, or a cost parameter for a machine learning architecture.

9. A system comprising:

one or more memory devices comprising a machine learning architecture comprising a plurality of tunable parameters, observed hyper-parameter sets, observed accuracy metrics, and observed training efficiency metrics; and

one or more computing devices that are configured to cause the system to:

generate a hyper-parameter set that balances accuracy and efficiency of learning the tunable parameters by:

fitting an accuracy regression model to pairs of the observed hyper-parameter sets and associated observed accuracy metrics;

fitting an efficiency regression model to pairs of the observed hyper-parameter sets and associated observed training efficiency metrics; and

selecting hyper-parameters for the hyper-parameter set by combining the accuracy regression model and the efficiency regression model utilizing an accuracy-training efficiency balance metric;

learn the plurality of tunable parameters of the machine learning architecture utilizing the hyper-parameter set to determine new accuracy metrics and new training efficiency metrics associated with the hyper-parameter set; and

select a modified hyper-parameter set based on the new accuracy metrics and the new training efficiency metrics.

10. The system of claim 9 , wherein the one or more computing devices are further configured to cause the system to select the modified hyper-parameter set by:

modifying the accuracy regression model to fit to the new accuracy metrics; and

modifying the efficiency regression model to fit the new training efficiency metrics.

11. The system of claim 10 , wherein the one or more computing devices are further configured to cause the system to select the modified hyper-parameter set by utilizing a trade-off acquisition function to weight the modified accuracy regression model relative to the modified efficiency regression model utilizing the accuracy-training efficiency balance metric.

12. The system of claim 11 , wherein the one or more computing devices are further configured to cause the system to determine the trade-off acquisition function by:

determining an accuracy acquisition function utilizing the accuracy regression model;

determining an efficiency acquisition function utilizing the efficiency regression model; and

combining the accuracy acquisition function and the efficiency acquisition function utilizing the accuracy-training efficiency balance metric.

13. The system of claim 9 , wherein the one or more computing devices are further configured to cause the system to:

identify an additional accuracy-training efficiency balance metric; and

generate an additional hyper-parameter set utilizing by combining the accuracy regression model and the efficiency regression model utilizing the additional accuracy-training efficiency balance metric.

14. The system of claim 13 , wherein the one or more computing devices are configured to cause the system to:

modify the plurality of tunable parameters of the machine learning architecture to determine additional accuracy metrics and additional training efficiency metrics associated with the additional hyper-parameter set; and

compare the additional accuracy metrics and the additional training efficiency metrics with the new accuracy metrics and the new training efficiency metrics.

15. The system of claim 9 , wherein the hyper-parameter set comprises one or more of a training set size, a dropout rate, a regularization parameter, a hidden unit, a batch size, a number of epochs, a bias, or a cost parameter for the machine learning architecture.

16. A computer-implemented method for selecting hyper-parameters for training parametrized machine learning models, the computer-implemented method comprising:

identifying observed hyper-parameter sets for a machine learning architecture comprising a plurality of tunable parameters;

generating a hyper-parameter set that balances accuracy and efficiency of learning the tunable parameters by:

fitting an accuracy regression model to pairs of the observed hyper-parameter sets and associated accuracy metrics;

fitting an efficiency regression model to pairs of the observed hyper-parameter sets and associated training efficiency metrics; and

selecting hyper-parameters for the hyper-parameter set utilizing a weighted combination of the accuracy regression model and the efficiency regression model; and

learning the tunable parameters of the machine learning architecture utilizing the hyper-parameter set.

17. The computer-implement method of claim 16 , wherein the machine learning architecture is part of one of a support vector machine or a neural network.

18. The computer-implemented method of claim 16 , further comprising determining observed accuracy metrics and observed training efficiency metrics associated with the observed hyper-parameter sets.

19. The computer-implemented method of claim 18 , further comprising determining new accuracy metrics and new training efficiency metrics associated with the hyper-parameter set based on performance of the machine learning architecture.

20. The computer-implemented method of claim 16 , wherein the hyper-parameter set comprises a training set size.