Systems and methods for implementing an intelligent machine learning optimization platform for multiple tuning criteria
Systems and methods for tuning hyperparameters of a model includes: receiving a multi-criteria tuning work request for tuning hyperparameters of the model of the subscriber to the remote tuning service, wherein the multi-criteria tuning work request includes: a first objective function of the model to be optimized by the remote tuning service; a second objective function to be optimized by the remote tuning service, the second objective function being distinct from the first objective function; computing a joint tuning func-tion based on a combination of the first objective function and the second objective function; executing a tuning opera-tion of the hyperparameters for the model based on a tuning of the joint function; and identifying one or more proposed hyperparameter values based on one or more hyperparam-eter-based points along a convex Pareto optimal curve.
1 . A system comprising:
interface circuitry to access a tuning work request; and
programmable circuitry to tune original hyperparameters of a prediction machine learning (ML) model based on the tuning work request, the programmable circuitry to:
access user input associated with the tuning work request, the user input specifying performance metrics of the prediction ML model;
perform, in parallel, hyperparameter optimization on a plurality of candidate models based on a budget associated with the user input, the hyperparameter optimization to map hyperparameter values of the plurality of candidate models to the performance metrics;
obtain a plurality of hyperparameter values by executing a plurality of hyperparameter optimization models, the plurality of hyperparameter values corresponding to respective prediction accuracies and prediction speeds of candidate prediction models utilizing respective ones of the plurality of hyperparameter optimization models;
compute, based on an objective function, a plurality of combinations of the respective prediction accuracies and prediction speeds of the respective ones of the plurality of hyperparameter values;
select one of the plurality of hyperparameter values based on a comparison of the combinations, a prediction accuracy of the prediction ML model, and a prediction speed of the prediction ML model; and
update the prediction ML model to replace the original hyperparameter values with the selected one of the plurality of hyperparameter values.
2 . The system of claim 1 , wherein the programmable circuitry includes a plurality of distributed computers, the distributed computers to perform the tuning of the hyperparameters.
3 . The system of claim 1 , wherein the programmable circuitry implements a tuning service.
4 . The system of claim 1 , wherein the ML model is at least one of a random search model, a Tree Parzen Estimation (TPE) model, or a reinforcement learning based model.
5 . The system of claim 1 , wherein the programmable circuitry is to determine a joint function that facilitates simultaneous optimization of a first accuracy metric and a first prediction speed metric.
6 . The system of claim 1 , wherein the programmable circuitry is to change a tuning value to increase an emphasis of the optimization of a first accuracy metric and also decrease an emphasis of a first prediction speed metric.
7 . The system of claim 1 , wherein the programmable circuitry is to cause presentation via a display of a first accuracy metric and a first prediction speed metric corresponding to a first candidate model and a second accuracy metric and a second prediction speed metric corresponding to a second candidate model.
8 . The system of claim 1 , wherein the interface circuitry is an application programming interface.
9 . The system of claim 1 , wherein the programmable circuitry is to select a first hyperparameter values based on calculation utilizing two dimensional objective functions in which the accuracy is a first dimension and the prediction speed is a second dimension.
10 . The system of claim 1 , wherein the programmable circuitry is to perform the hyperparameter optimization on the plurality of candidate models based on the budget by dividing the budget among three phases, wherein the phases include three different sweeps.
11 . The system of claim 10 , wherein the three different sweeps are: a quasi-random sweep of hyperparameter values, a sweep of groups of hyperparameter values, and a search of hyperparameter values that maximizes one objective at a time.
12 . The system of claim 11 , wherein a portion of the budget allocated to the sweep of groups of hyperparameter values is greater than portions of the budget allocated to the quasi-random sweep of hyperparameter values and the search of hyperparameter values that maximizes one objective at a time.
13 . A computer-readable storage device or storage disk comprising instructions to cause programmable circuitry to at least:
access user input associated with a tuning work request, the user input specifying performance metrics of a prediction machine learning (ML) model;
perform, in parallel, hyperparameter optimization on a plurality of candidate models based on a budget associated with the user input, the hyperparameter optimization to map hyperparameter values of the plurality of candidate models to the performance metrics;
obtain a plurality of hyperparameter values by executing a plurality of hyperparameter optimization models, the plurality of hyperparameter values corresponding to respective prediction accuracies and prediction speeds of candidate prediction models utilizing respective ones of the plurality of hyperparameter optimization models;
compute, based on an objective function, a plurality of combinations of the respective prediction accuracies and prediction speeds of the respective ones of the plurality of hyperparameter values;
select one of the plurality of hyperparameter values based on a comparison of the combinations, a prediction accuracy of the prediction ML model, and a prediction speed of the prediction ML model; and
update the prediction ML model to replace the original hyperparameter values with the selected one of the plurality of hyperparameter values.
14 . The computer-readable storage device or storage disk of claim 13 , wherein the instructions are to cause the programmable circuitry to perform the hyperparameter optimization using a plurality of distributed computers.
15 . The computer-readable storage device or storage disk of claim 13 , wherein the instructions are to cause the programmable circuitry to implement a tuning service.
16 . The computer-readable storage device or storage disk of claim 13 , wherein the ML model is at least one of a random search model, a Tree Parzen Estimation (TPE) model, or a reinforcement learning based model.
17 . The computer-readable storage device or storage disk of claim 13 , wherein the instructions are to cause the programmable circuitry to determine a joint function to facilitate simultaneous optimization of a first accuracy metric and a first prediction speed metric.
18 . The computer-readable storage device or storage disk of claim 13 , wherein the instructions are to cause the programmable circuitry to change a tuning value to increase an emphasis of the optimization of a first accuracy metric and also decrease an emphasis of a first prediction speed metric.
19 . A method to tune hyperparameters of a prediction machine learning (ML) model based on a tuning work request, the method comprising:
accessing user input associated with the tuning work request, the user input specifying performance metrics of the prediction ML model;
performing, by executing an instruction with programmable circuitry, hyperparameter optimization on a plurality of candidate models, in parallel, based on a budget associated with the user input, the hyperparameter optimization to map hyperparameter values of the plurality of candidate models to the performance metrics;
obtaining, by executing an instruction with the programmable circuitry, a plurality of hyperparameter values by executing a plurality of hyperparameter optimization models, the plurality of hyperparameter values corresponding to respective prediction accuracies and prediction speeds of candidate prediction models utilizing respective ones of the plurality of hyperparameter optimization models;
compute, by executing an instruction with the programmable circuitry, based on an objective function, a plurality of combinations of the respective prediction accuracies and prediction speeds of the respective ones of the plurality of hyperparameter values;
select one of the plurality of hyperparameter values based on a comparison of the combinations, a prediction accuracy of the prediction ML model, and a prediction speed of the prediction ML model; and
update, by executing an instruction with the programmable circuitry, the prediction ML model to replace the original hyperparameter values with the first selected one of the plurality of hyperparameter values.
20 . The method of claim 19 , wherein the hyperparameter optimization is performed by a plurality of distributed computers.