Systems and methods for business analytics model scoring and selection
The present invention relates to systems and methods for model scoring and selection. Six or more metrics that are relevant to the model are initially selected, and weights are assigned to each metric. A first subset of the metrics are selected, including metrics for model fit and model error for primary regression. A second subset of metrics including at least two penalty functions are then selected for percentage of incidence. The scores from the primary regression and penalty calculations are aggregated into a final score. Multiple models can be scored and utilized to select a “best” model via an iterative culling of low scoring models and “breeding” of the high scoring models.
1 . A computerized method for selecting a model, useful in association with a business analytics system, the method comprising:
receiving, at a computer system, an initial set of models, wherein the models comprise a series of variables representable as a binary string;
scoring, in the computer system, each of the initial set of models, wherein the scoring includes predicting model error using an iterative regression model of millions of forecast simulations;
ranking, in the computer system, the initial set of models by their scores;
removing, in the computer system, a subset of the initial set of models with a ranking below a threshold to yield a remaining set of models;
randomly selecting, in the computer system, a subset of the variables;
exchanging, in the computer system, the binary strings associated with the randomly selected subset of variables between the remaining set of models to generate a new set of models;
scoring, in the computer system, each of the new set of models;
determining if the new set of models is acceptable;
when the new set of models is not acceptable then ranking the new set of models by their scores, removing a subset of the new set of models with a ranking below a threshold to yield a new remaining set of models, and
repeating the prior four steps in the computer system;
when the new set of models is acceptable, selecting, in the computer system, a model from the new set of models with the highest score.
2 . The method of claim 1 , further comprising selecting a random variable, and at least one of removing or altering the selected random variable when exchanging the binary strings.
3 . The method of claim 1 , wherein the determining if the new set of models is acceptable includes comparing the scores of the new set of models against scores from prior scoring iteration.
4 . The method of claim 3 , wherein the determining if the new set of models is acceptable is when the scores of the new set of models have changed less that a threshold as compared against the scores from the prior scoring iteration.
5 . The method of claim 1 , wherein the determining if the new set of models is acceptable is when the scores of the new set of models are all above a threshold.
6 . The method of claim 1 , wherein the scoring of the initial set of models and new set of models includes at least a primary regression analysis and a penalty function.
7 . The method of claim 1 , wherein the subset of models removed are models with a score below a required threshold.
8 . The method of claim 1 , wherein the subset of models removed are half of the set of models with the lowest ranking.
9 . The method of claim 1 , further comprising storing the scoring of each model in a shared database such that each model is only scored once.
10 . The method of claim 1 , further comprising removing variables from the series of variables with a p-value above a threshold.
11 . A computerized system for selecting a model comprising:
a model generator for providing an initial set of models, wherein the models comprise a series of variables representable as a binary string;
a model scoring server for scoring each of the initial set of models, wherein the scoring includes predicting model error using an iterative regression model of millions of forecast simulations;
a model quality assessment server for ranking the initial set of models by their scores, removing a subset of the initial set of models with a ranking below a threshold to yield a remaining set of models, randomly selecting a subset of the variables, and exchanging the binary strings associated with the randomly selected subset of variables between the remaining set of models to generate a new set of models;
the model scoring server for scoring each of the new set of models; and
the model quality assessment server for determining if the new set of models is acceptable, and when the new set of models is not acceptable then ranking the new set of models by their scores, removing a subset of the new set of models with a ranking below a threshold to yield a new remaining set of models, and in conjunction with the model scoring server, iterating the scoring, ranking, and determining steps, and when the new set of models is acceptable, selecting a model from the new set of models with the highest score.
12 . The system of claim 11 , wherein the model quality assessment server is further configured for selecting a random variable, and at least one of removing or altering the selected random variable when exchanging the binary strings.
13 . The system of claim 11 , wherein the determining if the new set of models is acceptable includes comparing the scores of the new set of models against scores from prior scoring iteration.
14 . The system of claim 13 , wherein the determining if the new set of models is acceptable is when the scores of the new set of models have changed less that a threshold as compared against the scores from the prior scoring iteration.
15 . The system of claim 11 , wherein the determining if the new set of models is acceptable is when the scores of the new set of models are all above a threshold.
16 . The system of claim 11 , wherein the scoring of the initial set of models and new set of models includes at least a primary regression analysis and a penalty function.
17 . The system of claim 11 , wherein the subset of models removed are models with a score below a required threshold.
18 . The system of claim 11 , wherein the subset of models removed are half of the set of models with the lowest ranking.
19 . The system of claim 11 , further comprising storing the scoring of each model in a shared database such that each model is only scored once.
20 . The system of claim 11 , further comprising removing variables from the series of variables with a p-value above a threshold.