System and method for processing a large data set using a prediction model having a feature selection capability
View Patent ↗The present invention includes a system and a method for processing large data sets that are too large to conveniently fit into a formal database application. The large data set processing system and method use a prediction model having a feature selection capability to process a fraction of the large data set and define useful predictors. The useful features are used to make predictions for the entire data set. The large data set processing system includes a useful predictor module, for defining useful predictors, and a feature-selection prediction model, for processing a portion of the data set (including the useful predictors) to obtain prediction results.
1. A computer-implemented method for processing a data set containing samples and features, comprising:
loading a portion of the samples and each of the features into a database application;
building a feature-selection prediction model using each of the features and a portion of the samples loaded in the database application;
defining useful predictors using the feature-selection prediction model, the useful predictors corresponding to a portion of the features;
loading the useful predictors and each of the samples into a database application; and
determining prediction results by processing the feature-selection prediction model against the samples and useful predictors contained in the database application;
wherein the data set is too large for all of the samples and all of the features to be loaded into the database application simultaneously.
2. The computer-implemented method of claim 1 , wherein the feature-selection prediction model is a decision tree.
3. The computer-implemented method of claim 1 , further comprising obtaining a desired prediction for use by the feature-selection prediction model.
4. The computer-implemented method of claim 3 , wherein defining useful predictors further comprises determining whether a feature is useful and relevant in determining the desired prediction.
5. The computer-implemented method of claim 4 , wherein a feature is defined as a useful predictor if is useful and relevant in determining the desired prediction.
6. The computer-implemented method of claim 4 , further comprising ranking each feature based how useful and relevant the feature is in determining the desired prediction.
7. The computer-implemented method of claim 3 , wherein the prediction results are results obtained from making the desired prediction about each of the samples.
8. A computer-implemented method for processing a large set of data containing samples and features, the large data set being unable to fit into a database application, comprising:
building a prediction model having a feature selection capability using each of the features and a randomly-selected fraction of the samples;
using the feature-selection prediction model to define a portion of the features as useful predictors based on how helpful each feature is in making a desired prediction; ranking each feature based how useful and relevant the feature is in determining the desired prediction, wherein using the feature-selection prediction model to define a portion of the features as useful predictors further comprises determining whether a predicted probabilty for the desired prediction is greater tha a threshold probability; and
processing each of the samples and the useful predictors using the feature-selection prediction model to determine prediction results for the desired prediction.
9. The computer-implemented method of claim 8 , further comprising determining that the desired prediction is positive if the predicted probability greater than the threshold probability.
10. The computer-implemented method of claim 8 , further comprising determining that the desired prediction is negative if the predicted probability less than the threshold probability.
11. The computer-implemented method of claim 8 , further comprising loading the randomly-selected fraction of the samples and each of the features into the database application.
12. The computer-implemented method of claim 8 , further comprising loading each of the samples and each of the useful predictors into the database application.
13. A computer-readable medium having computer-executable instructions for processing a data set containing samples and features that cannot be loaded into a database application, comprising:
loading the first subset and the second subset into the database application;
obtaining a first subset containing a fraction of the data set including each of the features;
building a feature-selection prediction model using the first subset;
defining useful predictors using the feature-selection prediction model, the useful predictors corresponding to features that are helpful and useful in making a desired prediction;
obtaining a second subset containing a fraction of the data set including those features defined as useful predictors; and
processing the second subset using the feature-selection prediction model to obtain prediction results of the desired prediction for each of the samples.
14. The computer-readable medium of claim 13 , wherein the first subset further contains a randomly-selected fraction of the samples.
15. The computer-readable medium of claim 13 , wherein the second subset further contains each of the samples.
16. The computer-readable medium of claim 13 , further comprising ranking each of the useful predictors based on its usefulness in making the desired prediction.
17. The computer-readable medium of claim 16 , further comprising selecting a portion of the useful predictors based on their ranking.