Estimation of predictive accuracy gains from added features
View Patent ↗Various technologies described herein pertain to estimating predictive accuracy gain of a potential feature added to a set of features, wherein an existing predictor is trained on the set of features. Outputs of the existing predictor for instances in a dataset can be retrieved from a data store. Moreover, a predictive accuracy gain estimate of a potential feature added to the set of features can be measured as a function of the outputs of the existing predictor for the instances in the dataset. The predictive accuracy gain estimate can be measured without training an updated predictor on the set of features augmented by the potential feature.
1. A method executed by one or more computer processors, the method comprising:
receiving, from data storage, outputs of an existing predictor for instances in a dataset, the existing predictor is trained on a set of features;
computing a predictive accuracy gain estimate of a potential feature, the predictive accuracy gain estimate indicative of a potential contribution if the potential feature were to be added to the set of features, wherein:
the set of features on which the existing predictor is trained lacks the potential feature;
the predictive accuracy gain estimate is computed without training an updated predictor on the set of features augmented by the potential feature; and
the predictive accuracy gain estimate of the potential feature is computed based on:
the outputs of the existing predictor for the instances in the dataset;
labels for the instances in the dataset, the labels are target outputs being predicted for the instances in the dataset; and
values of the potential feature for the instances in the dataset;
identifying whether the potential feature is a candidate for inclusion in the set of features based on the predictive accuracy gain estimate of the potential feature, the potential feature being identified as one of a candidate or not a candidate for inclusion in the set of features without retraining of the predictor in a task-independent manner; and
outputting whether the potential feature is a candidate for inclusion in the set of features.
2. The method of claim 1 , further comprising:
training an incremental predictor based on the outputs of the existing predictor for the instances in the dataset, the labels for the instances in the dataset, and the values of the potential feature for the instances in the dataset; and
evaluating performance of the incremental predictor to compute the predictive accuracy gain estimate of the potential feature.
3. The method of claim 2 , wherein the incremental predictor is trained to predict proxy values of a loss gradient of the existing predictor for the instances in the dataset based on the values of the potential feature for the instances in the dataset.
4. The method of claim 3 , wherein the proxy values of the loss gradient comprise values of a pseudo gradient.
5. The method of claim 3 , wherein the proxy values of the loss gradient comprise values of a gradient estimate.
6. The method of claim 3 , further comprising computing a correlation between outputs of the incremental predictor and the proxy values of the loss gradient.
7. The method of claim 2 , wherein the incremental predictor is trained to predict values of a loss gradient of the existing predictor for the instances in the dataset based on the values of the potential features for the instances in the dataset.
8. The method of claim 1 , wherein the predictive accuracy gain estimate of the potential feature is computed without access to the existing predictor.
9. The method of claim 1 , wherein the predictive accuracy gain estimate of the potential feature is computed without use of values of features in the set of features on which the existing predictor is trained.
10. The method of claim 1 , further comprising computing the predictive accuracy gain estimate of the potential feature based on a dependency of the potential feature to a residual of the existing predictor.
11. The method of claim 1 , further comprising:
computing a second predictive accuracy gain estimate of a second potential feature, the second predictive accuracy gain estimate indicative of a second potential contribution if the second potential feature were to be added to the set of features, wherein:
the set of features on which the existing predictor is trained lacks the second potential feature;
the second predictive accuracy gain estimate is computed without training an updated predictor on the set of features augmented by the second potential feature; and
the second predictive accuracy gain estimate of the second potential feature is computed based on:
the outputs of the existing predictor for the instances in the dataset;
the labels for the instances in the dataset; and
values of the second potential feature for the instances in the dataset; and
outputting, based on the second predictive accuracy gain estimate of the second potential feature, whether the second potential feature is a candidate for inclusion in the set of features.
12. The method of claim 1 , wherein the predictive accuracy gain estimate is computed for the potential feature and a second potential feature, the predictive accuracy gain estimate indicative of the potential contribution if the potential feature and the second potential feature were to be added to the set of features.
13. The method of claim 1 , further comprising:
computing a confidence score for whether the potential feature improves predictive accuracy; and
outputting the confidence score.
14. A system that evaluates a predictive accuracy gain estimate of a potential feature, comprising:
one or more processors; and
data storage, comprising:
outputs of an existing predictor for instances in a dataset, wherein the existing predictor is trained on a set of features that lacks the potential feature;
labels for the instances in the dataset; and
values of the potential feature for the instances in the dataset; and
the data storage further comprises computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts including:
training an incremental predictor based on the outputs of the existing predictor for the instances in the dataset, the labels for the instances in the dataset, and the values of the potential feature for the instances in the dataset;
computing the predictive accuracy gain estimate of the potential feature based on outputs of the incremental predictor, the predictive accuracy gain estimate indicative of a potential contribution if the potential feature were to be added to the set of features, and the predictive accuracy gain estimate is computed without an updated predictor being trained on the set of features augmented by the potential feature;
identifying whether the potential feature is a candidate for inclusion in the set of features based on the predictive accuracy gain estimate of the potential feature, the potential feature being identified as one of a candidate or not a candidate for inclusion in the set of features without retraining of the predictor in a task-independent manner; and
outputting whether the potential feature is a candidate for inclusion in the set of features.
15. The system of claim 14 , wherein the incremental predictor is trained to predict proxy values of a loss gradient of the existing predictor for the instances in the dataset based on the values of the potential feature for the instances in the dataset.
16. The system of claim 14 , wherein the predictive accuracy gain estimate is computed without access to the existing predictor.
17. The system of claim 14 , the data storage further comprises computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts including:
performing a test to determine a confidence score that signifies whether the potential feature improves a prediction of the existing predictor.
18. A system that evaluates a predictive accuracy gain estimate of a potential feature, comprising:
one or more processors; and data storage, comprising:
outputs of an existing predictor for instances in a dataset, wherein the existing predictor is trained on a set of features that lacks the potential feature;
labels for the instances in the dataset; and
values of the potential feature for the instances in the dataset; and
the data storage further comprises computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts including:
computing the predictive accuracy gain estimate of the potential feature based on the outputs of the existing predictor for the instances in the dataset, the labels for the instances in the dataset, and the values of the potential feature for the instances in the dataset, wherein the predictive accuracy gain estimate is computed without training an updated predictor on the set of features augmented by the potential feature;
identifying whether the potential feature is a candidate for inclusion in the set of features based on the predictive accuracy gain estimate of the potential feature, the potential feature being identified as one of a candidate or not a candidate for inclusion in the set of features without retraining of the predictor in a task-independent manner; and
outputting whether the potential feature is a candidate for inclusion in the set of features.
19. The system of claim 18 , wherein:
the predictive accuracy gain estimate of the potential feature is computed based on outputs of an incremental predictor; and
the incremental predictor is trained based on the outputs of the existing predictor for the instances in the dataset, the labels for the instances in the dataset, and the values of the potential feature for the instances in the dataset without being trained on the set of features on which the existing predictor is trained.
20. The system of claim 19 , wherein the incremental predictor is trained to predict proxy values of a loss gradient of the existing predictor for the instances in the dataset based on the values of the potential feature for the instances in the dataset.