Dynamically tuning hyperparameters during ML model training
A method of automatically tuning hyperparameters includes receiving a hyperparameter tuning strategy. Upon determining that one or more computing resources exceed their corresponding predetermined quota, the hyperparameter tuning strategy is rejected. Upon determining that the one or more computing resources do not exceed their corresponding predetermined quota, a machine learning model training is run with a hyperparameter point. Upon determining that one or more predetermined computing resource usage limits are exceeded for the hyperparameter point, the running of the machine learning model training is terminated for the hyperparameter point and the process returns to running the machine learning model training with a new hyperparameter point. Upon determining that training the machine learning model is complete, training results are collected and computing resource utilization metrics are determined. A correlation of the hyperparameters to the computing resource utilization is determined from the completed training of the machine learning model.
1 . A computing device, comprising:
a processor;
a network interface coupled to the processor to enable communication over a network;
a storage device for content and programming coupled to the processor; and
a hyperparameter engine stored in the storage device, wherein an execution of the hyperparameter engine by the processor configures the computing device to perform operations comprising:
receiving a first hyperparameter tuning strategy;
based on determining that one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, rejecting the first hyperparameter tuning strategy and receive a second hyperparameter tuning strategy different from the first hyperparameter tuning strategy,
wherein the determining, that the one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, is based on each hyperparameter point of the first hyperparameter tuning strategy being within an admissible region of a geometric convex hull in a hyperparameter space of the first hyperparameter tuning strategy;
based on determining that the one or more computing resources do not exceed their corresponding predetermined quota for the second hyperparameter tuning strategy, running a machine learning model training with a first hyperparameter point of the second hyperparameter tuning strategy;
monitoring, during each iteration of the machine learning model training, computing resource utilization for the first hyperparameter point;
based on determining that the computing resource utilization exceeds one or more predetermined computing resource usage limits for the first hyperparameter point:
terminating the running of the machine learning model training for the first hyperparameter point;
updating the admissible region based on the terminating of the machine learning model training, wherein the first hyperparameter point lies outside of the updated admissible region; and
running the machine learning model training with a second hyperparameter point, different from the first hyperparameter point, of the second hyperparameter tuning strategy;
based on determining that the machine learning model training is complete, collecting training results and determining computing resource utilization metrics;
learning, from the updated admissible region and the completion of the machine learning model training, a correlation of one or more hyperparameters of the second hyperparameter tuning strategy to the computing resource utilization; and
utilizing the learned correlation of the one or more hyperparameters to select hyperparameter points for subsequent machine leaning training runs.
2 . The computing device of claim 1 , wherein the first hyperparameter point of the second hyperparameter tuning strategy is selected randomly.
3 . The computing device of claim 1 , wherein the correlation of the one or more hyperparameters to the computing resource utilization is learned additionally based on one or more recorded previously successful completed machine learning model trainings.
4 . The computing device of claim 1 , wherein the correlation of the one or more hyperparameters to the computing resource utilization is learned based on a probe run.
5 . The computing device of claim 4 , wherein the probe run generates a multi-dimensional grid of the one or more hyperparameters of permissible hyperparameter combinations.
6 . The computing device of claim 1 , wherein the second hyperparameter tuning strategy is based on a grid search.
7 . The computing device of claim 1 , wherein the second hyperparameter tuning strategy is based on a random search.
8 . The computing device of claim 1 , wherein the running of the machine learning model training further comprises:
dividing a training data into predetermined batches; and
iteratively running each training batch of the predetermined batches.
9 . The computing device of claim 1 , wherein the correlation of the one or more hyperparameters to the computing resource utilization is learned by measuring an increasing or decreasing trend of the computing resource utilization metrics in a case where a hyperparameter of the one or more hyperparameters is changed in magnitude.
10 . The computing device of claim 1 , wherein the execution of the hyperparameter engine by the processor configures the computing device to perform the operations further comprising:
based on the determining that the computing resource utilization exceeds the one or more predetermined computing resource usage limits for the first hyperparameter point:
logging the termination of the machine learning model training for the first hyperparameter point;
updating the admissible region based on the logged termination of the machine learning model training; and
running the machine learning model training with the second hyperparameter point that lies within the updated admissible region, wherein
the admissible region corresponds to a region comprising one or more hyperparameter points of one or more hyperparameter tuning strategies that are feasible for the machine learning model training without exceeding a computational resource budget, and
the one or more hyperparameter tuning strategies includes the first hyperparameter tuning strategy and the second hyperparameter tuning strategy.
11 . A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computing device to carry out a method of automatically tuning hyperparameters, the method comprising:
receiving a first hyperparameter tuning strategy;
based on determining that one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, rejecting the first hyperparameter tuning strategy and receiving a second hyperparameter tuning strategy different from the first hyperparameter tuning strategy,
wherein the determining, that the one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, is based on each hyperparameter point of the first hyperparameter tuning strategy being within an admissible region of a geometric convex hull in a hyperparameter space of the first hyperparameter tuning strategy;
based on determining that the one or more computing resources do not exceed their corresponding predetermined quota for the second hyperparameter tuning strategy, running a machine learning model training with a first hyperparameter point of the second hyperparameter tuning strategy;
monitoring, during each iteration of the machine learning model training, computing resource utilization for the first hyperparameter point;
based on determining that the computing resource utilization exceeds one or more predetermined computing resource usage limits for the first hyperparameter point:
terminating the running of the machine learning model training for the first hyperparameter point;
updating the admissible region based on the termination of the machine learning model training, wherein the first hyperparameter point lies outside of the updated admissible region; and
running the machine learning model training with a second hyperparameter point, different from the first hyperparameter point, of the second hyperparameter tuning strategy;
based on determining that the machine learning model training is complete, collecting training results and determining computing resource utilization metrics;
learning, from the updated admissible region and the completion of the machine learning model training, a correlation of one or more hyperparameters of the second hyperparameter tuning strategy to the computing resource utilization; and
utilizing the learned correlation of the one or more hyperparameters to select hyperparameter points for subsequent machine learning training runs.
12 . The non-transitory computer readable storage medium of claim 11 , wherein the learning of the correlation of the one or more hyperparameters to the computing resource utilization is additionally based on one or more recorded previously successful completed machine learning model trainings.
13 . The non-transitory computer readable storage medium of claim 11 , wherein
the learning of the correlation of the one or more hyperparameters to the computing resource utilization is based on a probe run; and
the probe run generates a multi-dimensional grid of the one or more hyperparameters of permissible hyperparameter combinations.
14 . The non-transitory computer readable storage medium of claim 11 , wherein the second hyperparameter tuning strategy is based on a grid search.
15 . The non-transitory computer readable storage medium of claim 11 , wherein the second hyperparameter tuning strategy is based on a random search.
16 . The non-transitory computer readable storage medium of claim 11 , wherein the running of the machine learning model training further comprises:
dividing a training data into predetermined batches; and
iteratively running each training batch of the predetermined batches.
17 . A computer implemented method of automatically tuning hyperparameters, the computer implemented method comprising:
receiving a first hyperparameter tuning strategy;
based on determining that one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, rejecting the first hyperparameter tuning strategy and receiving a second hyperparameter tuning strategy different from the first hyperparameter tuning strategy,
wherein the determining, that the one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, is based on each hyperparameter point of the first hyperparameter tuning strategy being within an admissible region of a geometric convex hull in a hyperparameter space of the first hyperparameter tuning strategy;
based on determining that the one or more computing resources do not exceed their corresponding predetermined quota for the second hyperparameter tuning strategy, running a machine learning model training with a first hyperparameter point of the second hyperparameter tuning strategy;
monitoring, during each iteration of the machine learning model training, computing resource utilization for the first hyperparameter point;
based on determining that the computing resource utilization exceeds one or more predetermined computing resource usage limits for the first hyperparameter point:
terminating the running of the machine learning model training for the first hyperparameter point;
updating the admissible region based on the termination of the machine learning model training, wherein the first hyperparameter point lies outside of the updated admissible region; and
running the machine learning model training with a second hyperparameter point, different from the first hyperparameter point, of the second hyperparameter tuning strategy;
based on determining that the machine learning model training is complete, collecting training results and determining computing resource utilization metrics;
learning, from the updated admissible region and the completion of the machine learning model training, a correlation of one or more hyperparameters of the second hyperparameter tuning strategy to the computing resource utilization; and
utilizing the learned correlation of the one or more hyperparameters to select hyperparameter points for subsequent machine learning training runs.
18 . The computer implemented method of claim 17 , wherein the learning of the correlation of the one or more hyperparameters to the computing resource utilization is additionally based on one or more recorded previously successful completed machine learning model trainings.
19 . The computer implemented method of claim 17 , wherein
the learning of the correlation of the one or more hyperparameters to the computing resource utilization is based on a probe run; and
the probe run generates a multi-dimensional grid of the one or more hyperparameters of permissible hyperparameter combinations.