Time-bound hyperparameter tuning

1 . A method comprising:

determining a total time available (T avail ) for training a machine learning (ML) model using a training dataset, the training comprising performing hyperparameter optimization for the ML model;

executing a number (N initial ) of initial trials, wherein executing the number of initial trials comprises, for each trial in the initial trials:

for each hyperparameter in a set of hyperparameters, selecting a hyperparameter value for the hyperparameter from a full search space associated with the hyperparameter, and

training the ML model using the training dataset and the selected set of hyperparameter values;

determining an approximate time for executing a trial (T approx ) based upon executing the initial trials;

using a first computer-implemented technique to determine a first number (N search ) of first trials to be executed, wherein using the first computer-implemented technique comprises using the total time available (T avail ) and the approximate time for executing a trial (T approx );

executing the first number (N search ) of first trials, wherein executing the first number of first trials comprises, for each first trial in the first trials:

for each hyperparameter in the set of hyperparameters, selecting a hyperparameter value for the hyperparameter from a full search space associated with the hyperparameter, and

training the ML model using the training dataset and the selected set of hyperparameter values;

using a second computer-implemented technique to determine a second number (N reduced ) of second trials to be executed, wherein the second computer-implemented technique comprises uses N search ;

executing the second number (N reduced ) of second trials, wherein executing the second number of second trials comprises, for each second trial in the second trials:

for each hyperparameter in the set of hyperparameters, selecting a hyperparameter value for the hyperparameter from a reduced search space associated with the hyperparameter; and

training the ML model using the training dataset and the selected set of hyperparameter values selected from the reduced search space associated with each hyperparameter in the set of hyperparameters;

selecting a particular second trial from the second trials;

designating the hyperparameter values used for the particular second trial as a set of optimized hyperparameter values; and

outputting a trained ML model trained in the particular second trial and the set of optimized hyperparameter values.

2 . The method of claim 1 , wherein using the first computer-implemented technique comprises:

based upon the total time available (T avail ) for training the ML model and the approximate time for executing a trial, determining an approximate number of trials (N o );

determining a first Fibonacci number using N o ; and

determining the first number (N search ) of first trials by subtracting the number (N initial ) of initial trials from the first Fibonacci number.

3 . The method of claim 2 wherein determining the first Fibonacci number comprises:

determining n using the equation n(1+F 2 )=N o , wherein F is a ratio represented by

1

+

5

2

;

 and

identifying a Fibonacci number that is equal to or greater than nF 2 as the first Fibonacci number.

4 . The method of claim 2 , wherein using the second computer-implemented technique comprises:

determining a third Fibonacci number based upon the first Fibonacci number; and

designating the third Fibonacci number as the second number (N reduced ) of second trials.

5 . The method of claim 4 wherein determining the third Fibonacci number comprises:

identifying a sequence of three consecutive Fibonacci numbers, wherein the first Fibonacci number is the largest number in the sequence; and

designating the smallest Fibonacci number in the sequence as the third Fibonacci number.

6 . The method of claim 4 further comprising:

selecting a number (N top ) of top trials from the initial trials and the first trials;

based upon the selected top trials, for at least one hyperparameter in the set of hyperparameters, identifying a reduced search space for the at least one hyperparameter, where the identified reduced search space for the at least one hyperparameter has a smaller number of values than a number of values in the full search space for the at least one hyperparameter; and

wherein executing the second number (N reduced ) of second trials comprises, for each second trial, setting a value for the at least one hyperparameter from the reduced search space identified for the at least one hyperparameter.

7 . The method of claim 6 wherein selecting the number (N top ) of top trials from the initial trials and the first trials comprises:

determining a second Fibonacci number based upon the first Fibonacci number; and

designating the second Fibonacci number as N top .

8 . The method of claim 7 wherein determining the second Fibonacci number comprises designating a Fibonacci number that immediately precedes the first Fibonacci number as the second Fibonacci number.

9 . The method of claim 6 wherein identifying the reduced search space for the at least one hyperparameter comprises:

determining a highest value used for the at least one hyperparameter in the selected top trials;

determining a lowest value used for the at least one hyperparameter in the selected top trials; and

determining a range of values wherein the range is bounded by the highest value and the lowest value, wherein the range represents the reduced search space for the at least one hyperparameter.

10 . The method of claim 4 further comprising:

selecting a number (N top ) of top trials from the initial trials and the first trials;

based upon the selected top trials:

for at least one hyperparameter in the set of hyperparameters, identifying a reduced search space for the hyperparameter, where the identified reduced search space for the hyperparameter has a smaller number of values then a number of values in the full search space for the at least one hyperparameter; and

identifying a reduced set of features from a set of features used in the initial trials and in the first trials;

wherein executing the second number (N reduced ) of second trials comprises, for each second trial:

setting a value for the at least one hyperparameter from the reduced search space identified for the at least one hyperparameter; and

using the reduced set of features.

11 . The method of claim 10 wherein identifying the reduced set of features comprises:

selecting one or more features, from the set of features, to be included in the reduced set of features based upon feature importance scores assigned to the set of features.

12 . The method of claim 1 , wherein, for at least one trial in the initial trials, first trials, or second trials, Bayesian optimization is used to select values for the hyperparameters in the set of hyperparameters.

13 . The method of claim 1 , wherein determining the approximate time for executing a trial comprises:

determining an execution time taken for executing each of the initial trials;

determining an average time by averaging the execution times determined for executing each of the initial trials; and

designating the average time as the approximate time for executing a trial.

14 . The method of claim 1 , wherein the training dataset comprises one or more time series, each time series comprising a plurality of datapoints, each data point in the plurality of datapoints characterized by a time and at least one associated value.

15 . The method of claim 1 , wherein selecting the particular second trial from the second trials comprises selecting a trial from the second trials having a highest performance metric.

16 . The method of claim 1 , wherein different combinations of hyperparameter values are used for the initial trials, the first trials, and the second trials.

17 . The method of claim 1 , wherein the training dataset comprises a training portion, a validation portion, and a test portion, wherein the validation portion is used during hyperparameter optimization.

18 . A system comprising:

a set of processors;

a memory storing a set of optimized hyperparameter values for a trained machine learning (“ML”) model, wherein determining the set of optimized hyperparameter values comprises:

determining a total time available (T avail ) for training a machine learning (ML) model using a training dataset, the training comprising performing hyperparameter optimization for the ML model;

executing a number (N initial ) of initial trials, wherein executing the number of initial trials comprises, for each trial in the initial trials:

for each hyperparameter in a set of hyperparameters, selecting a hyperparameter value for the hyperparameter from a full search space associated with the hyperparameter, and

training the ML model using the training dataset and the selected set of hyperparameter values;

determining an approximate time for executing a trial (T approx ) based upon executing the initial trials;

using a first computer-implemented technique to determine a first number (N search ) of first trials to be executed, wherein using the first computer-implemented technique comprises using the total time available (T avail ) and the approximate time for executing a trial (T approx );

executing the first number (N search ) of first trials, wherein executing the first number of first trials comprises, for each first trial in the first trials:

for each hyperparameter in the set of hyperparameters, selecting a hyperparameter value for the hyperparameter from a full search space associated with the hyperparameter, and

training the ML model using the training dataset and the selected set of hyperparameter values;

using a second computer-implemented technique to determine a second number (N reduced ) of second trials to be executed, wherein the second computer-implemented technique comprises uses N search ;

executing the second number (N reduced ) of second trials, wherein executing the second number of second trials comprises, for each second trial in the second trials:

for each hyperparameter in the set of hyperparameters, selecting a hyperparameter value for the hyperparameter from a reduced search space associated with the hyperparameter; and

training the ML model using the training dataset and the selected set of hyperparameter values selected from the reduced search space associated with each hyperparameter in the set of hyperparameters;

selecting a particular second trial from the second trials;

designating the hyperparameter values used for the particular second trial as a set of optimized hyperparameter values; and

outputting a trained ML model trained in the particular second trial and the set of optimized hyperparameter values.

19 . The system of claim 18 , wherein:

using the first computer-implemented technique comprises:

based upon the total time available (T avail ) for training the ML model and the approximate time for executing a trial, determine an approximate number of trials (N o );

determining a first Fibonacci number using N o ; and

determining the first number (N search ) of first trials by subtracting the number (N initial ) of initial trials from the first Fibonacci number; and

using the second computer-implemented technique comprises:

determining a third Fibonacci number based upon the first Fibonacci number; and

designating the third Fibonacci number as the second number (N reduced ) of second trials.

20 . A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more computer devices, cause the computing devices to perform processing comprising:

determining a total time available (T avail ) for training a machine learning (ML) model using a training dataset, the training comprising performing hyperparameter optimization for the ML model;

executing a number (N initial ) of initial trials, wherein executing the number of initial trials comprises, for each trial in the initial trials:

for each hyperparameter in a set of hyperparameters, selecting a hyperparameter value for the hyperparameter from a full search space associated with the hyperparameter, and

training the ML model using the training dataset and the selected set of hyperparameter values;

determining an approximate time for executing a trial (T approx ) based upon executing the initial trials;

using a first computer-implemented technique to determine a first number (N search ) of first trials to be executed, wherein using the first computer-implemented technique comprises using the total time available (T avail ) and the approximate time for executing a trial (T approx );

executing the first number (N search ) of first trials, wherein executing the first number of first trials comprises, for each first trial in the first trials:

for each hyperparameter in the set of hyperparameters, selecting a hyperparameter value for the hyperparameter from a full search space associated with the hyperparameter, and

training the ML model using the training dataset and the selected set of hyperparameter values;

using a second computer-implemented technique to determine a second number (N reduced ) of second trials to be executed, wherein the second computer-implemented technique comprises uses N search ;

executing the second number (N reduced ) of second trials, wherein executing the second number of second trials comprises, for each second trial in the second trials:

for each hyperparameter in the set of hyperparameters, selecting a hyperparameter value for the hyperparameter from a reduced search space associated with the hyperparameter; and

training the ML model using the training dataset and the selected set of hyperparameter values selected from the reduced search space associated with each hyperparameter in the set of hyperparameters;

selecting a particular second trial from the second trials;

designating the hyperparameter values used for the particular second trial as a set of optimized hyperparameter values; and

outputting a trained ML model trained in the particular second trial and the set of optimized hyperparameter values.