Optimizing feature importance for binary classification
Feature importance is critical to understanding how predictive models produce accurate results, and can change significantly for different models. The present invention is used to achieve a good ranking for stable feature importance. An optimized technique is presented which considers feature importance value variation within different groups of cross-trained models. Feature importance is computed for all group models with this optimized method, and then a best set of models can be selected based on classification error as well as optimized stable feature importance values.
1 . A computer-implemented method of optimizing feature importance for binary classification comprising:
receiving a plurality of binary classification models;
generating multiple confidence threshold values;
building multiple model groups, each model group including all of the binary classification models for a different one of the confidence threshold values;
splitting training data into multiple parts and forming different training data sets from the parts;
cross-training all models within each model group using the different training data sets;
computing an individual classification error for each model in each model group;
merging individual classification errors for all models within a particular model group to arrive at a combined classification error for each model group;
selecting a set of the model groups based on their respective combined classification errors relative to an error threshold;
calculating a feature importance value for each feature in each model of a given model group in the set;
computing an importance stability value for each model group in the set based on the feature importance values;
selecting a final model group from the set having the maximum importance stability value; and
selecting a plurality of optimized features based on a ranking of feature importance for the final model group.
2 . The computer-implemented method of claim 1 wherein the multiple confidence threshold values are generated by incrementing a starting confidence threshold value by a step value.
3 . The computer-implemented method of claim 2 wherein the starting confidence threshold value is based on a ratio of positive-to-negative cases in the training data.
4 . The computer-implemented method of claim 1 wherein a given training data set excludes at least one part of the training data, and a given individual classification error is computed by testing each model using a respective one of the excluded parts of the training data.
5 . The computer-implemented method of claim 1 wherein said computing the importance stability value for the given model group includes:
computing intermediate stability values for different pairs of models in the given model group according to a number of changed feature importance values between models in a given pair; and
summing the intermediate stability values of all pairs in the given model group.
6 . The computer-implemented method of claim 1 wherein there is more than one final model group having the maximum importance stability value, and further comprising selecting a particular one of the final model groups based on a voting scheme that uses means and variances of the feature importance values for models in each of the final model groups.
7 . The computer-implemented method of claim 1 further comprising:
training a binary classifier according to the final model group and the optimized features; and
using the binary classifier to compute a most likely category for input scenario data.
8 . A computer system comprising:
one or more processors which process program instructions;
a memory device connected to said one or more processors; and
program instructions residing in said memory device for optimizing feature importance for binary classification by receiving a plurality of binary classification models, generating multiple confidence threshold values, building multiple model groups, each model group including all of the binary classification models for a different one of the confidence threshold values, splitting training data into multiple parts and forming different training data sets from the parts, cross-training all models within each model group using the different training data sets, computing an individual classification error for each model in each model group, merging individual classification errors for all models within a particular model group to arrive at a combined classification error for each model group, selecting a set of the model groups based on their respective combined classification errors relative to an error threshold, calculating a feature importance value for each feature in each model of a given model group in the set, computing an importance stability value for each model group in the set based on the feature importance values, selecting a final model group from the set having the maximum importance stability value, and selecting a plurality of optimized features based on a ranking of feature importance for the final model group.
9 . The computer system of claim 8 wherein the multiple confidence threshold values are generated by incrementing a starting confidence threshold value by a step value.
10 . The computer system of claim 8 wherein the starting confidence threshold value is based on a ratio of positive-to-negative cases in the training data.
11 . The computer system of claim 8 wherein a given training data set excludes at least one part of the training data, and a given individual classification error is computed by testing each model using a respective one of the excluded parts of the training data.
12 . The computer system of claim 8 wherein computing the importance stability value for the given model group includes:
computing intermediate stability values for different pairs of models in the given model group according to a number of changed feature importance values between models in a given pair; and
summing the intermediate stability values of all pairs in the given model group.
13 . The computer system of claim 8 wherein there is more than one final model group having the maximum importance stability value, and further comprising selecting a particular one of the final model groups based on a voting scheme that uses means and variances of the feature importance values for models in each of the final model groups.
14 . The computer system of claim 8 wherein said program instructions further:
train a binary classifier according to the final model group and the optimized features; and
use the binary classifier to compute a most likely category for input scenario data.
15 . A computer program product comprising:
one or more computer readable storage media; and
program instructions collectively residing in said one or more computer readable storage media for optimizing feature importance for binary classification by receiving a plurality of binary classification models, generating multiple confidence threshold values, building multiple model groups, each model group including all of the binary classification models for a different one of the confidence threshold values, splitting training data into multiple parts and forming different training data sets from the parts, cross-training all models within each model group using the different training data sets, computing an individual classification error for each model in each model group, merging individual classification errors for all models within a particular model group to arrive at a combined classification error for each model group, selecting a set of the model groups based on their respective combined classification errors relative to an error threshold, calculating a feature importance value for each feature in each model of a given model group in the set, computing an importance stability value for each model group in the set based on the feature importance values, selecting a final model group from the set having the maximum importance stability value, and selecting a plurality of optimized features based on a ranking of feature importance for the final model group.
16 . The computer program product of claim 15 wherein the multiple confidence threshold values are generated by incrementing a starting confidence threshold value by a step value.
17 . The computer program product of claim 15 wherein the starting confidence threshold value is based on a ratio of positive-to-negative cases in the training data.
18 . The computer program product of claim 15 wherein a given training data set excludes at least one part of the training data, and a given individual classification error is computed by testing each model using a respective one of the excluded parts of the training data.
19 . The computer program product of claim 15 wherein computing the importance stability value for the given model group includes:
computing intermediate stability values for different pairs of models in the given model group according to a number of changed feature importance values between models in a given pair; and
summing the intermediate stability values of all pairs in the given model group.
20 . The computer program product of claim 15 wherein there is more than one final model group having the maximum importance stability value, and further comprising selecting a particular one of the final model groups based on a voting scheme that uses means and variances of the feature importance values for models in each of the final model groups.