DYNAMIC MODEL DATA FACILITY AND AUTOMATED OPERATIONAL MODEL BUILDING AND USAGE
A commercial process with a dependent variable can be associated with a set of independent variables. The commercial process can continuously provide data collection opportunities. An intervention is designed using a model to predict the dependent outcome. The actual outcome of the intervention can be determined within the window of utility for these data. One objective is to improve intervention outcomes with prediction. Purely random outcomes (no model prediction) and outcomes resulting from the intervention (model operations) are aggregated into separate files—a sequence of control model data files and a sequence of model data files of operational data. These model data files and control model data files are used to analyze model performance and to react automatically when identified conditions warrant.
1 . A method, comprising:
building a model;
using the model in a first expert system;
obtaining at least some ongoing operational data from the first expert system while the first expert system uses the model; and
repeatedly testing the model against actual data to determine whether or not to recalibrate or re-specify the model.
2 . The method of claim 1 , wherein a second expert system performs the testing of the model.
3 . The method of claim 2 , wherein the second expert system obtains random samples of data from the first expert system to test the model.
4 . The method of claim 1 , further comprising:
obtaining an independent source of random data; and
using the independent source of random data as part of the testing.
5 . The method of claim 1 , wherein the model is tested by tracking a goodness-of-fit (GOF).
6 . The method of claim 5 , further comprising:
gathering one or more statistics relevant to calibration for the actual data;
gathering one or more statistics relevant to discrimination for the actual data;
comparing the one or more statistics relevant to calibration to a first pre-existing validation data;
comparing the one or more statistics relevant to discrimination to a second pre-existing validation data; and
based on the comparisons, determining the GOF for the model.
7 . A system, comprising:
a processor; and
a computer readable medium coupled to the processor and comprising instructions that, when executed by the processor, enable the processor to perform the following:
build a model;
use the model in a first expert system;
obtain at least some ongoing operational data from the first expert system while the first expert system uses the model; and
repeatedly test the model against actual data to determine whether or not to recalibrate or re-specify the model.
8 . The system of claim 7 , wherein a second expert system performs the testing of the model.
9 . The system of claim 8 , wherein the second expert system obtains random samples of data from the first expert system to test the model.
10 . The system of claim 7 , wherein the instructions further enable the processor perform the following:
obtain an independent source of random data; and
use the independent source of random data as part of the testing.
11 . The system of claim 7 , wherein the model is tested by tracking a goodness-of-fit (GOF).
12 . The method of claim 11 , wherein the instructions further enable the processor perform the following:
gather one or more statistics relevant to calibration for the actual data;
gather one or more statistics relevant to discrimination for the actual data;
compare the one or more statistics relevant to calibration to a first pre-existing validation data;
compare the one or more statistics relevant to discrimination to a second pre-existing validation data; and
based on the comparisons, determine the GOF for the model.
13 . A computer learning system, comprising:
one or more servers capable of building a model, using the model in a first expert system, obtaining at least some ongoing operational data from the first expert system while the first expert system uses the model, and repeatedly testing the model against actual data to determine whether or not to recalibrate or re-specify the model.
14 . The computer learning system of claim 13 , wherein a second expert system performs the testing of the model.
15 . The computer learning system of claim 14 , wherein the second expert system obtains random samples of data from the first expert system to test the model.
16 . The computer learning system of claim 13 , wherein the one or more servers are further capable of obtaining an independent source of random data and using the independent source of random data as part of the testing.
17 . The method of claim 13 , wherein the model is tested by tracking a goodness-of-fit (GOF).
18 . The method of claim 17 , wherein the one or more servers are further capable of gathering one or more statistics relevant to calibration for the actual data, gathering one or more statistics relevant to discrimination for the actual data, comparing the one or more statistics relevant to calibration to a first pre-existing validation data, comparing the one or more statistics relevant to discrimination to a second pre-existing validation data, and, based on the comparisons, determining the GOF for the model.