Feature importance based model optimization
Disclosed are a computer-implemented method, a system and a computer program product for model exploration. Model feature importance of each model of a plurality of models can be obtained, the plurality of models can be grouped into a plurality of model clusters based on the model feature importance of each model, and the model feature importance can be presented by box-plot or confidence interval.
1. A computer-implemented method for
exploring predictive models, the method comprising:
computing feature importance for each of a plurality of features for each of a plurality of predictive models;
clustering predictive models among the plurality of predictive models to form a plurality of model clusters based on a similarity of feature importance of the computed feature importance for each of the plurality of features for each of the plurality of predictive models;
deriving a cluster feature importance for each model cluster of the plurality of model clusters;
computing a feature importance for a data case using a preliminary model;
selecting a model cluster for the data case from the plurality of model clusters based on similarity between the feature importance for the data case and the cluster feature importance of each model cluster of the plurality of model clusters; and
selecting a predictive model within the selected model cluster to score the data case based on at least one criteria thereby optimizing an evaluation of the plurality of predictive models among a massive set of predictive models.
2. The computer-implemented method of claim 1 ,
wherein the preliminary model is selected based on at least one of a plurality of model attributes from the plurality of predictive models to be explored.
3. The computer-implemented method of claim 2 , wherein the plurality of model attributes comprise model performance, model size, model accuracy, and model complexity.
4. The computer-implemented method of claim 1 ,
further comprising:
performing a model interpretation algorithm on the preliminary model to deduce importance values for features of the data case to compute the feature importance for the data case using the preliminary model.
5. The computer-implemented method of claim 1 ,
wherein the feature importance is computed for each of the plurality of predictive models using sensitivity analysis.
6. The computer-implemented method of claim 1 ,
wherein the computed feature importance for each of the plurality of features for each of the plurality of predictive models is displayed via a box-plot or a confidence interval to show a distribution of feature importance among the plurality of predictive models.
7. The computer-implemented method of claim 1 ,
wherein the at least one criteria comprises one or more of the following in the group consisting of a model type, a model size, a model accuracy, and a model performance.
8. The computer-implemented method of claim 1 , further comprising:
clustering predictive models among the plurality of predictive models to form the plurality of model clusters using a k-means based algorithm.
9. The computer-implemented method of claim 1 ,
wherein the similarity between the feature importance for the data case and the cluster feature importance of each model cluster of the plurality of model clusters is computed via a measured distance between vectors.
10. A computer program product for exploring predictive models, the computer program product comprising one or more computer readable storage mediums having program code embodied therewith, the program code comprising programming instructions for:
computing feature importance for each of a plurality of features for each of a plurality of predictive models;
clustering predictive models among the plurality of predictive models to form a plurality of model clusters based on a similarity of feature importance of the computed feature importance for each of the plurality of features for each of the plurality of predictive models;
deriving a cluster feature importance for each model cluster of the plurality of model clusters;
computing a feature importance for a data case using a preliminary model;
selecting a model cluster for the data case from the plurality of model clusters based on similarity between the feature importance for the data case and the cluster feature importance of each model cluster of the plurality of model clusters; and
selecting a predictive model within the selected model cluster to score the data case based on at least one criteria thereby optimizing an evaluation of the plurality of predictive models among a massive set of predictive models.
11. The computer program product of claim 10 , wherein the preliminary model is selected based on at least one of a plurality of model attributes from the plurality of predictive models to be explored.
12. The computer program product of claim 11 , wherein the plurality of model attributes comprise model performance, model size, model accuracy, and model complexity.
13. The computer program product of claim 10 , wherein the program code further comprises the programming instructions for:
performing a model interpretation algorithm on the preliminary model to deduce importance values for features of the data case to compute the feature importance for the data case using the preliminary model.
14. The computer program product of claim 10 , wherein the feature importance is computed for each of the plurality of predictive models using sensitivity analysis.
15. The computer program product of claim 10 , wherein the computed feature importance for each of the plurality of features for each of the plurality of predictive models is displayed via a box-plot or a confidence interval to show a distribution of feature importance among the plurality of predictive models.
16. The computer program product of claim 10 , wherein the at least one criteria comprises one or more of the following in the group consisting of a model type, a model size, a model accuracy, and a model performance.
17. The computer program product of claim 10 , wherein the program code further comprises the programming instructions for:
clustering predictive models among the plurality of predictive models to form the plurality of model clusters using a k-means based algorithm.
18. The computer program product of claim 10 , wherein the similarity between the feature importance for the data case and the cluster feature importance of each model cluster of the plurality of model clusters is computed via a measured distance between vectors.
19. A system, comprising:
a memory for storing a computer program for exploring predictive models; and
a processor connected to the memory, wherein the processor is configured to execute program instructions of the computer program comprising:
computing feature importance for each of a plurality of features for each of a plurality of predictive models;
clustering predictive models among the plurality of predictive models to form a plurality of model clusters based on a similarity of feature importance of the computed feature importance for each of the plurality of features for each of the plurality of predictive models;
deriving a cluster feature importance for each model cluster of the plurality of model clusters;
computing a feature importance for a data case using a preliminary model;
selecting a model cluster for the data case from the plurality of model clusters based on similarity between the feature importance for the data case and the cluster feature importance of each model cluster of the plurality of model clusters; and
selecting a predictive model within the selected model cluster to score the data case based on at least one criteria thereby optimizing an evaluation of the plurality of predictive models among a massive set of predictive models.
20. The system of claim 19 , wherein the preliminary model is selected based on at least one of a plurality of model attributes from the plurality of predictive models to be explored.
21. The system of claim 20 , wherein the plurality of model attributes comprise model performance, model size, model accuracy, and model complexity.
22. The system of claim 19 , wherein the program instructions of the computer program further comprise:
performing a model interpretation algorithm on the preliminary model to deduce importance values for features of the data case to compute the feature importance for the data case using the preliminary model.
23. The system of claim 19 , wherein the feature importance is computed for each of the plurality of predictive models using sensitivity analysis.
24. The system of claim 19 , wherein the computed feature importance for each of the plurality of features for each of the plurality of predictive models is displayed via a box-plot or a confidence interval to show a distribution of feature importance among the plurality of predictive models.
25. The system of claim 19 , wherein the at least one criteria comprises one or more of the following in the group consisting of a model type, a model size, a model accuracy, and a model performance.