Systems and methods for multi-factor model selection and promotion
A model selection method includes: receiving a request to train or validate a plurality of models where the request includes training data and one or more trigger conditions; obtaining the models from a model catalog and training the models using the training data to obtain results for each of the one or more trigger conditions; and selecting, based on the results of each of the one or more trigger conditions and from among the models, a best model to be pushed to production.
1 . A model selection method comprising:
receiving a request to train or validate a plurality of machine learning models, wherein the request comprising training data and one or more trigger conditions, wherein the trigger conditions specify;
model accuracy;
model latency;
model size;
amount of computing resources required for training; and
requirements and restrictions;
obtaining the machine learning models from a model catalog;
training the machine learning models using the training data to obtain results for each of the one or more trigger conditions;
selecting, based on the results of each of the one or more trigger conditions, on the requirements and restrictions, and from among the machine learning models, a best model to be pushed to production, wherein the requirements and restrictions specify lower model latency should be prioritized over model accuracy and that model accuracy should be at least 90%;
updating, in response to the selecting, the model catalog, wherein updating the model catalog comprises:
removing the best model and all data associated with the best model from the model catalog; and
ranking non-selected machine learning models of the model catalog based on the results of the selecting;
obtaining, after the updating, a plurality of hardware configurations and testing the best model on each hardware configuration within the plurality of hardware configurations to obtain configuration results, wherein the configuration results are the results of the pairing of the best model and each hardware configuration;
selecting, based on the configuration results and from the plurality of hardware configurations, a best hardware configuration; and
implementing, based on selecting the best hardware configuration, the best model using the best hardware configuration.
2 . The model selection method of claim 1 , wherein the training data comprises ground-truth data.
3 . A system comprising:
a memory; and
a processor coupled to the memory, wherein the processor is configured to execute a model selection method comprising:
receiving a request to train or validate a plurality of machine learning models, wherein the request comprising training data and one or more trigger conditions, wherein the trigger conditions specify;
model accuracy;
model latency;
model size;
amount of computing resources required for training; and
requirements and restrictions;
obtaining the machine learning models from a model catalog;
training the machine learning models using the training data to obtain results for each of the one or more trigger conditions;
selecting, based on the results of each of the one or more trigger conditions, on the restrictions, on the requirements, and from among the machine learning models, a best model to be pushed to production, wherein the requirements specify lower model latency should be prioritized over model accuracy and that model accuracy should be at least 90%;
updating, in response to the selecting, the model catalog, wherein updating the model catalog comprises:
removing the best model and all data associated with the best model from the model catalog; and
ranking non-selected machine learning models of the model catalog based on the results of the selecting;
obtaining, after the selecting, a plurality of hardware configurations and testing the best model on each hardware configuration within the plurality of hardware configurations to obtain configuration results, wherein the configuration results are the results of the pairing of the best model and each hardware configuration;
selecting, based on the configuration results and from the plurality of hardware configurations, a best hardware configuration; and
implementing, based on selecting the best hardware configuration, the best model using the best hardware configuration.
4 . The system of claim 3 , wherein the training data comprises ground-truth data.