Automatic feature subset selection based on meta-learning
The present invention relates to dimensionality reduction for machine learning (ML) models. Herein are techniques that individually rank features and combine features based on their rank to achieve an optimal combination of features that may accelerate training and/or inferencing, prevent overfitting, and/or provide insights into somewhat mysterious datasets. In an embodiment, a computer ranks features of datasets of a training corpus. For each dataset and for each landmark percentage, a target ML model is configured to receive only a highest ranking landmark percentage of features, and a landmark accuracy achieved by training the ML model with the dataset is measured. Based on the landmark accuracies and meta-features values of the dataset, a respective training tuple is generated for each dataset. Based on all of the training tuples, a regressor is trained to predict an optimal amount of features for training the target ML model.
1. A method comprising:
ranking a plurality of features of a plurality of datasets;
for each dataset of the plurality of datasets:
for each percentage of a plurality of distinct percentages:
configuring a machine learning (ML) model to receive only a highest ranking said percentage of the plurality of features, and
measuring an accuracy, of a respective plurality of accuracies, achieved by training the ML model with the dataset; and
generating, based on a plurality of meta-features values of the dataset, a respective tuple of a plurality of training tuples that contains the respective plurality of accuracies; and
training, based on the plurality of training tuples that contain respective pluralities of accuracies, a regressor that predicts a count of features of the plurality of features to configure the ML model to be most accurate.
2. The method of claim 1 further comprising:
receiving a new dataset;
for each percentage of the plurality of distinct percentages:
configuring the ML model to receive only a highest ranking said percentage of the plurality of features, and
measuring a new accuracy, of a new plurality of accuracies, achieved by training the ML model with the new dataset; and
generating, based on the new plurality of accuracies and a new plurality of meta-features values of the new dataset, a new tuple;
predicting, by the regressor and based on the new tuple, a new count of features of the plurality of features.
3. The method of claim 2 further comprising:
configuring the ML model to receive only a highest ranking said new count of features of the plurality of features;
measuring an empirical accuracy achieved by training the ML model with the new dataset and said highest ranking said new count of features;
selecting a subset of the plurality of features based on the empirical accuracy and the new plurality of accuracies.
4. The method of claim 3 wherein a size of the plurality of features does not affect complexity of
said selecting the subset of the plurality of features.
5. The method of claim 1 wherein said predicts the count of features of the plurality of features comprises:
predicts a percentage of the plurality of features, or
predicts an optimal subset of the plurality of features.
6. The method of claim 1 wherein:
said ranking the plurality of features comprises a plurality of rankings of the plurality of features;
said for each dataset comprises for each dataset and each particular ranking of the plurality of rankings;
said highest ranking said percentage of the plurality of features comprises said highest ranking by the particular ranking;
said predicts said count of features of the plurality of features comprises predicts a respective count of features for each particular ranking of the plurality of rankings.
7. The method of claim 1 wherein the regressor comprises a random forest.
8. The method of claim 1 wherein a count of the plurality of datasets exceeds one hundred.
9. The method of claim 1 wherein the plurality of meta-features values of each dataset of the plurality of datasets comprises at least one selected from the group consisting of:
a count of the plurality of features that are numeric,
a count of the plurality of features that are not numeric,
a ratio of the count of the plurality of features to the count of samples in the dataset,
a count of classes of samples in the dataset that the ML model can recognize,
a minority count of samples in the dataset of a least frequent class of said classes,
a majority count of samples in the dataset of a most frequent class of said classes, and
a ratio of the minority count to the majority count.
10. The method of claim 1 wherein the accuracy comprises: a precision, a recall, or an F score.
11. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause:
ranking a plurality of features of a plurality of datasets;
for each dataset of the plurality of datasets:
for each percentage of a plurality of distinct percentages:
configuring a machine learning (ML) model to receive only a highest ranking said percentage of the plurality of features, and
measuring an accuracy, of a respective plurality of accuracies, achieved by training the ML model with the dataset; and
generating, based on a plurality of meta-features values of the dataset, a respective tuple of a plurality of training tuples that contains the respective plurality of accuracies; and
training, based on the plurality of training tuples that contain respective pluralities of accuracies, a regressor that predicts a count of features of the plurality of features to configure the ML model to be most accurate.
12. The one or more non-transitory computer-readable storage media of claim 11 wherein the instructions further cause:
receiving a new dataset;
for each percentage of the plurality of distinct percentages:
configuring the ML model to receive only a highest ranking said percentage of the plurality of features, and
measuring a new accuracy, of a new plurality of accuracies, achieved by training the ML model with the new dataset; and
generating, based on the new plurality of accuracies and a new plurality of meta-features values of the new dataset, a new tuple;
predicting, by the regressor and based on the new tuple, a new count of features of the plurality of features.
13. The one or more non-transitory computer-readable storage media of claim 12 wherein the instructions further cause:
configuring the ML model to receive only a highest ranking said new count of features of the plurality of features;
measuring an empirical accuracy achieved by training the ML model with the new dataset and said highest ranking said new count of features;
selecting a subset of the plurality of features based on the empirical accuracy and the new plurality of accuracies.
14. The one or more non-transitory computer-readable storage media of claim 13 wherein a size of the plurality of features does not affect complexity of
said selecting the subset of the plurality of features.
15. The one or more non-transitory computer-readable storage media of claim 11 wherein said predicts the count of features of the plurality of features comprises:
predicts a percentage of the plurality of features, or
predicts an optimal subset of the plurality of features.
16. The one or more non-transitory computer-readable storage media of claim 11 wherein:
said ranking the plurality of features comprises a plurality of rankings of the plurality of features;
said for each dataset comprises for each dataset and each particular ranking of the plurality of rankings;
said highest ranking said percentage of the plurality of features comprises said highest ranking by the particular ranking;
said predicts said count of features of the plurality of features comprises predicts a respective count of features for each particular ranking of the plurality of rankings.
17. The one or more non-transitory computer-readable storage media of claim 11 wherein
the regressor comprises a random forest.
18. The one or more non-transitory computer-readable storage media of claim 11 wherein a count of the plurality of datasets exceeds one hundred.
19. The one or more non-transitory computer-readable storage media of claim 11 wherein the plurality of meta-features values of each dataset of the plurality of datasets comprises at least one selected from the group consisting of:
a count of the plurality of features that are numeric,
a count of the plurality of features that are not numeric,
a ratio of the count of the plurality of features to the count of samples in the dataset,
a count of classes of samples in the dataset that the ML model can recognize,
a minority count of samples in the dataset of a least frequent class of said classes,
a majority count of samples in the dataset of a most frequent class of said classes, and
a ratio of the minority count to the majority count.
20. The one or more non-transitory computer-readable storage media of claim 11 wherein the accuracy comprises: a precision, a recall, or an F score.