System and method for a personalized machine learning pipeline selection and result interpretation
A method for managing data includes obtaining a request for a machine learning (ML) pipeline selection from a client, wherein the request comprises a training dataset and a domain of the training dataset, and in response to the request: identifying a set of ML pipelines based on the domain, obtaining runtime statistics for the set of ML pipelines using the domain and at least a portion the training dataset, generating, using a user preference model, an ordering of the set of ML pipelines based on the runtime statistics and user preferences, and presenting the ordering, the runtime statistics, and a notification based on the ordering to the client.
1. A method for managing data, the method comprising:
obtaining, by a machine learning (ML) execution environment, a request for a ML pipeline selection from a client, wherein the request comprises a training dataset and a domain of the training dataset; and
in response to the request:
identifying a set of ML pipelines based on the domain;
generating, by an execution node of the ML execution environment, runtime statistics for the set of ML pipelines using the domain and at least a portion of the training dataset, wherein generating the runtime statistics comprises:
performing a training on one of the set of ML pipelines using the at least a portion of the training dataset to generate a ML model;
executing the ML model on the training dataset; and
tracking performance of the ML model and of the training, wherein the runtime statistics specifies the performance;
generating, using a user preference model, a first ordering of the set of ML pipelines based on the runtime statistics and user preferences; and
presenting the first ordering, the runtime statistics, and a first notification based on the first ordering to the client;
obtaining, in response to the presenting, a ML selection that specifies a selected ML pipeline of the set of ML pipelines;
providing the selected ML pipeline to the client;
updating the user preferences based on the ML selection to obtain updated user preferences;
after obtaining the selected ML pipeline, executing the selected ML pipeline in a ML execution environment to generate a trained machine learning model;
after updating the user preferences:
obtaining a second request for a second ML pipeline selection from the client, wherein the second request comprises the training dataset and the domain of the training dataset;
in response to the second request:
generating, using the user preference model, a second ordering of the set of ML pipelines based on the runtime statistics and the updated user preferences;
presenting the second ordering, the runtime statistics, and a second notification based on the second ordering to the client; and
obtaining, in response to the presenting the second ordering, the runtime statistics, and the second notification based on the second ordering to the client, a second selection that specifies a second selected ML pipeline of the set of ML pipelines.
2. The method of claim 1 , wherein obtaining the runtime statistics for the set of ML pipelines comprises:
providing inputs of each ML pipeline in the set of ML pipelines into a prediction model to generate the runtime statistics.
3. The method of claim 2 , wherein the inputs comprise at least one of: a size of the training dataset, a type of ML algorithm associated with each ML pipeline, a number of hyper-parameters of each ML pipeline, and a standard dataset.
4. The method of claim 2 , wherein the runtime statistics comprise a plurality of criteria: and wherein the criteria comprise at least one of: an accuracy, a training cost, a training speed, an inferred cost and an inferred speed of each ML pipeline.
5. The method of claim 2 , further comprising:
obtaining ML pipeline telemetry associated with execution of the selected ML pipeline; and
updating the prediction model based on the ML pipeline telemetry.
6. The method of claim 5 , wherein the ML pipeline telemetry is obtained from a ML pipeline execution environment in which the selected ML pipeline is executing.
7. A system, comprising:
a processor; and
memory comprising instructions which, when executed by the processor, perform a method, the method comprising:
obtaining a request for a machine learning (ML) pipeline selection from a client, wherein the request comprises a training dataset and a domain of the training dataset; and
in response to the request:
identifying a set of ML pipelines based on the domain;
generating runtime statistics for the set of ML pipelines using the domain and at least a portion of the training dataset, wherein generating the runtime statistics comprises:
performing a training on one of the set of ML pipelines using the at least a portion of the training dataset to generate a ML model;
executing the ML model on the training dataset; and
tracking performance of the ML model and of the training, wherein the runtime statistics specifies the performance;
inputting the runtime statistics and user preferences associated with the client into a user preference model to obtain a first ordering of the set of ML pipelines; and
presenting the first ordering, the runtime statistics, and a notification based on the first ordering to the client;
obtaining, in response to the presenting, a ML selection that specifies a selected ML pipeline of the set of ML pipelines;
providing the selected ML pipeline to the client;
updating the user preferences based on the ML selection to obtain updated user preferences;
after obtaining the selected ML pipeline, executing the selected ML pipeline in a ML execution environment to generate a trained machine learning model;
after updating the user preferences:
obtaining a second request for a second ML pipeline selection from the client, wherein the second request comprises the training dataset and the domain of the training dataset; and
in response to the second request:
generating, using the user preference model, a second ordering of the set of ML pipelines based on the runtime statistics and the updated user preferences;
presenting the second ordering, the runtime statistics, and a notification based on the second ordering to the client; and
obtaining, in response to the presenting the first ordering, the runtime statistics, and a notification based on the second ordering to the client, a second selection that specifies a second selected ML pipeline of the set of ML pipelines.
8. The system of claim 7 , wherein obtaining the runtime statistics for the set of ML pipelines comprises:
providing inputs of each ML pipeline in the set of ML pipelines into a prediction model to generate the runtime statistics.
9. The system of claim 8 , wherein the inputs comprise at least one of: a size of the training dataset, a type of ML algorithm associated with each ML pipeline, a number of hyper-parameters of each ML pipeline, and a standard dataset.
10. The system of claim 8 , wherein the runtime statistics comprise a plurality of criteria: and wherein the criteria comprise at least one of: an accuracy, a training cost, a training speed, an inferred cost and an inferred speed of each ML pipeline.
11. The system of claim 8 , further comprising:
obtaining ML pipeline telemetry associated with execution of the selected ML pipeline; and
updating the prediction model based on the ML pipeline telemetry.
12. The system of claim 11 , wherein the ML pipeline telemetry is obtained from a ML pipeline execution environment in which the selected ML pipeline is executing.
13. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing data, the method comprising:
obtaining a request for a machine learning (ML) pipeline selection from a client, wherein the request comprises a training dataset and a domain of the training dataset; and
in response to the request:
identifying a set of ML pipelines based on the domain;
generating runtime statistics for the set of ML pipelines using the domain and at least a portion of the training dataset, wherein generating the runtime statistics comprises:
performing a training on one of the set of ML pipelines using the at least a portion of the training dataset to generate a ML model;
executing the ML model on the training dataset; and
tracking performance of the ML model and of the training, wherein the runtime statistics specifies the performance;
inputting the runtime statistics and user preferences associated with the client into a user preference model to obtain a first ordering of the set of ML pipelines; and
presenting the first ordering, the runtime statistics, and a notification based on the first ordering to the client;
obtaining, in response to the presenting a ML selection that specifies a selected ML pipeline of the set of ML pipelines;
providing the selected ML pipeline to the client;
updating the user preferences based on the ML selection to obtain updated user preferences;
after obtaining the selected ML pipeline, initiating, by the client, execution of the ML pipeline in a ML execution environment to generate a trained machine learning model;
after updating the user preferences:
obtaining a second request for a second ML pipeline selection from a client, wherein the second request comprises the training dataset and the domain of the training dataset;
in response to the request:
generating, using the user preference model, a second ordering of the set of ML pipelines based on the runtime statistics and the updated user preferences;
presenting the first ordering, the runtime statistics, and a notification based on the second ordering to the client; and
obtaining, in response to the presenting the first ordering, the runtime statistics, and a notification based on the second ordering to the client, a second selection that specifies a second selected ML pipeline of the set of ML pipelines.
14. The non-transitory computer readable medium of claim 13 , wherein obtaining the runtime statistics for the set of ML pipelines comprises:
providing inputs of each ML pipeline in the set of ML pipelines into a prediction model to generate the runtime statistics.
15. The non-transitory computer readable medium of claim 14 , wherein the inputs comprise at least one of: a size of the training dataset, a type of ML algorithm associated with each ML pipeline, a number of hyper-parameters of each ML pipeline, and a standard dataset.
16. The non-transitory computer readable medium of claim 14 , wherein the runtime statistics comprise a plurality of criteria: and wherein the criteria comprise at least one of: an accuracy, a training cost, a training speed, an inferred cost and an inferred speed of each ML pipeline.
17. The non-transitory computer readable medium of claim 14 , further comprising:
obtaining ML pipeline telemetry associated with execution of the selected ML pipeline; and
updating the prediction model based on the ML pipeline telemetry.