Large-scale automated hyperparameter tuning
Systems and methods determine optimized hyperparameter values for one or more machine-learning models. A sample training data set from a larger corpus of training data is obtained. Initial hyperparameter values are then randomly selected. Using the sample training data set and the randomly chosen hyperparameter values, an initial set of performance metric values are obtained. Maximized hyperparameter values are then determined from the initial set of hyperparameter values based on the corresponding performance metric value. A larger corpus of training data is then evaluated using the maximized hyperparameter values and the corresponding machine-learning model, which yields another corresponding set of performance metric values. The maximized hyperparameter values and their corresponding set of performance metric values are then merged with the prior set of hyperparameter values. The foregoing operations are performed iteratively until it is determined that the hyperparameter values are converging to a particular value.
1. A system comprising:
a computer-readable storage device storing computer-executable instructions; and
at least one hardware processor communicatively coupled to the computer-readable storage device that, when the computer-executable instructions are executed, configures the system to:
sample a predetermined amount of training data from a corpus of training data to obtain sampled training data;
select a predetermined number of hyperparameter values from a predetermined set of hyperparameter values to obtain an initial plurality of hyperparameter values, wherein each hyperparameter value of the initial plurality of hyperparameter values corresponds to at least one hyperparameter that defines a machine-learning model;
evaluate the machine-learning model using the sampled training data and the initial plurality of hyperparameter values to obtain a plurality of corresponding performance metric values;
assign the initial plurality of hyperparameter values and their corresponding performance metric values as a baseline set of hyperparameter values;
assign the baseline set of hyperparameter values as a current set of hyperparameter values;
iteratively perform the following operations until the at least one hardware processor determines that at least one hyperparameter is converging on a particular hyperparameter value:
determine a maximized plurality of hyperparameter values based on the current set of hyperparameter values;
determine a corresponding performance metric value for each hyperparameter value of the maximized plurality of hyperparameter values based on applying the machine-learning model to the corpus of training data using at least one hyperparameter value of the maximized plurality of hyperparameter values;
merge the maximized plurality of hyperparameter values and their corresponding performance metric values with the baseline set of values to obtain a merged set of hyperparameter values; and
assign the merged set of hyperparameter values as the current set of hyperparameter values on which to perform the determination and merging operations; and
transmit the maximized plurality of hyperparameter values as optimal hyperparameter values for corresponding hyperparameters of the machine-learning model.
2. The system of claim 1 , wherein the at least one hardware processor is further configured to:
determine a baseline bias value based on at least one performance metric value selected from the performance metric values associated with the baseline set of hyperparameter values; and wherein
the assignment of the baseline set of hyperparameter values as a current set of hyperparameter values comprises discounting at least one hyperparameter value of the baseline set of hyperparameter values by the determined baseline bias value.
3. The system of claim 1 , wherein the at least one hardware processor is further configured to:
determine a current bias value based on a number of iterations performed and at least one performance metric value selected from the performance metric values associated with the maximized plurality of hyperparameter values; and wherein
at least one hyperparameter of the maximized plurality of hyperparameter values is discounted by the determined current bias value.
4. The system of claim 1 , wherein the at least one hardware processor is further configured to:
estimate an integrated acquisition function based on at least one hyperparameter value selected from the current set of hyperparameter values and on at least one performance metric value associated with the at least one hyperparameter value; and wherein
the maximized plurality of hyperparameter values are further determined based on the estimated integrated acquisition function.
5. The system of claim 1 , wherein the at least one hardware processor is further configured to receive a selection of the at least one hyperparameter that defines the machine-learning model.
6. The system of claim 1 , wherein:
evaluation of the machine-learning model using the sampled training data and the initial plurality of hyperparameter values to obtain a plurality of corresponding performance metric values further comprises receiving a selection of a performance metric for which the plurality of corresponding performance metric values are obtained.
7. The system of claim 1 , wherein:
evaluation of the machine-learning model using the sampled training data and the initial plurality of hyperparameter values to obtain a plurality of corresponding performance metric values is performed in parallel within a distributed computing architecture.
8. A method comprising:
sampling, by at least one hardware processor, a predetermined amount of training data from a corpus of training data to obtain sampled training data;
selecting, by the at least one hardware processor, a predetermined number of hyperparameter values from a predetermined set of hyperparameter values to obtain an initial plurality of hyperparameter values, wherein each hyperparameter value of the initial plurality of hyperparameter values corresponds to at least one hyperparameter that defines a machine-learning model;
evaluating, by the at least one hardware processor, the machine-learning model using the sampled training data and the initial plurality of hyperparameter values to obtain a plurality of corresponding performance metric values;
assigning, by the at least one hardware processor, the initial plurality of hyperparameter values and their corresponding performance metric values as a baseline set of hyperparameter values;
assigning, by the at least one hardware processor, the baseline set of hyperparameter values as a current set of hyperparameter values;
iteratively performing the following operations until the at least one hardware processor determines that at least one hyperparameter is converging on a particular hyperparameter value:
determining a maximized plurality of hyperparameter values based on the current set of hyperparameter values;
determining a corresponding performance metric value for each hyperparameter value of the maximized plurality of hyperparameter values based on applying the machine-learning model to the corpus of training data using at least one hyperparameter value of the maximized plurality of hyperparameter values;
merging the maximized plurality of hyperparameter values and their corresponding performance metric values with the baseline set of values to obtain a merged set of hyperparameter values; and
assigning the merged set of hyperparameter values as the current set of hyperparameter values on which to perform the determination and merging operations; and
transmitting the maximized plurality of hyperparameter values as optimal hyperparameter values for corresponding hyperparameters of the machine-learning model.
9. The method of claim 8 , further comprising:
determining a baseline bias value based on at least one performance metric value selected from the performance metric values associated with the baseline set of hyperparameter values; and wherein
the assignment of the baseline set of hyperparameter values as a current set of hyperparameter values comprises discounting at least one hyperparameter value of the baseline set of hyperparameter values by the determined baseline bias value.
10. The method of claim 8 , further comprising:
determining a current bias value based on a number of iterations performed and at least one performance metric value selected from the performance metric values associated with the maximized plurality of hyperparameter values; and wherein
at least one hyperparameter of the maximized plurality of hyperparameter values is discounted by the determined current bias value.
11. The method of claim 8 , further comprising:
estimating an integrated acquisition function based on at least one hyperparameter value selected from the current set of hyperparameter values and on at least one performance metric value associated with the at least one hyperparameter value; and wherein
the maximized plurality of hyperparameter values are further determined based on the estimated integrated acquisition function.
12. The method of claim 8 , wherein the at least one hardware processor is further configured to receive a selection of the at least one hyperparameter that defines the machine-learning model.
13. The method of claim 8 , wherein:
evaluating the machine-learning model using the sampled training data and the initial plurality of hyperparameter values to obtain a plurality of corresponding performance metric values further comprises receiving a selection of a performance metric for which the plurality of corresponding performance metric values are obtained.
14. The method of claim 8 , wherein:
evaluating the machine-learning model using the sampled training data and the initial plurality of hyperparameter values to obtain a plurality of corresponding performance metric values is performed in parallel within a distributed computing architecture.
15. A computer-readable storage device storing computer-executable instructions that, when executed by at least one hardware processor, causes a system to perform a plurality of operations comprising:
sampling a predetermined amount of training data from a corpus of training data to obtain sampled training data;
selecting a predetermined number of hyperparameter values from a predetermined set of hyperparameter values to obtain an initial plurality of hyperparameter values, wherein each hyperparameter value of the initial plurality of hyperparameter values corresponds to at least one hyperparameter that defines a machine-learning model;
evaluating the machine-learning model using the sampled training data and the initial plurality of hyperparameter values to obtain a plurality of corresponding performance metric values;
assigning the initial plurality of hyperparameter values and their corresponding performance metric values as a baseline set of hyperparameter values;
assigning the baseline set of hyperparameter values as a current set of hyperparameter values;
iteratively performing the following operations until the at least one hardware processor determines that at least one hyperparameter is converging on a particular hyperparameter value:
determining a maximized plurality of hyperparameter values based on the current set of hyperparameter values;
determining a corresponding performance metric value for each hyperparameter value of the maximized plurality of hyperparameter values based on applying the machine-learning model to the corpus of training data using at least one hyperparameter value of the maximized plurality of hyperparameter values;
merging the maximized plurality of hyperparameter values and their corresponding performance metric values with the baseline set of values to obtain a merged set of hyperparameter values; and
assigning the merged set of hyperparameter values as the current set of hyperparameter values on which to perform the determination and merging operations; and
transmitting the maximized plurality of hyperparameter values as optimal hyperparameter values for corresponding hyperparameters of the machine-learning model.
16. The computer-readable storage device of claim 15 , wherein:
the plurality of operations further comprise determining a baseline bias value based on at least one performance metric value selected from the performance metric values associated with the baseline set of hyperparameter values; and
the assignment of the baseline set of hyperparameter values as a current set of hyperparameter values comprises discounting at least one hyperparameter value of the baseline set of hyperparameter values by the determined baseline bias value.
17. The computer-readable storage device of claim 15 , wherein:
the plurality of operations further comprise determining a current bias value based on a number of iterations performed and at least one performance metric value selected from the performance metric values associated with the maximized plurality of hyperparameter values; and
at least one hyperparameter of the maximized plurality of hyperparameter values is discounted by the determined current bias value.
18. The computer-readable storage device of claim 15 , wherein:
the plurality of operations further comprise estimating an integrated acquisition function based on at least one hyperparameter value selected from the current set of hyperparameter values and on at least one performance metric value associated with the at least one hyperparameter value; and
the maximized plurality of hyperparameter values are further determined based on the estimated integrated acquisition function.
19. The computer-readable storage device of claim 15 , wherein:
evaluation of the machine-learning model using the sampled training data and the initial plurality of hyperparameter values to obtain a plurality of corresponding performance metric values further comprises receiving a selection of a performance metric for which the plurality of corresponding performance metric values are obtained.
20. The computer-readable storage device of claim 15 , wherein:
evaluation of the machine-learning model using the sampled training data and the initial plurality of hyperparameter values to obtain a plurality of corresponding performance metric values is performed in parallel within a distributed computing architecture.