Computer-implemented systems and methods for updating predictive models
Computer-implemented systems and methods for updating champion predictive models that operate within a production environment. A system and method can include evaluating the predictive model performance of a champion predictive model. Based upon an indication of decay of predictive model performance of the champion predictive model, a corrective action is performed to correct the performance of the champion predictive model.
1. A computer-implemented method of assessing a predictive model, comprising:
accessing, using one or more data processors, a predictive model for scoring transactional data and data associated with a time period;
generating, using the one or more data processors, one or more predictions using the predictive model and the data;
calculating, using the one or more data processors, a baseline performance metric for the predictive model based upon the predictions;
accessing, using the one or more data processors, new data associated with a new time period that is later than the time period;
generating, using the one or more data processors, one or more new data predictions using the predictive model and the new data;
calculating, using the one or more data processors, an updated performance metric for the predictive model based upon the new data predictions;
comparing, using the one or more data processors, the baseline performance metric and the updated performance metric to determine an indication of predictive ability decay for the predictive model; and
performing, using the one or more data processors, a corrective action for the predictive model when the indication of predictive ability decay reaches a threshold.
2. The method of claim 1 , wherein the predictive model is operating within a production environment, wherein performance of an additional predictive model in the production environment is assessed, and wherein a second corrective action is performed on the additional predictive model when predictive ability decay of the additional predictive model reaches the threshold.
3. The method of claim 1 , wherein the predictive model generates predictions for fraud detection, rate making, credit scoring, customer retention, customer lifetime value, customer attrition or churn, or marketing response.
4. The method of claim 1 , wherein the predictive ability decay is caused by evolution of a customer base or changes in predictability of data used by the predictive model.
5. The method of claim 1 , wherein the corrective action is performed by one or more servers or a single general purpose computer.
6. The method of claim 1 , wherein the predictive model is a neural network model, a logistic model, or a decision tree model.
7. The method of claim 1 , wherein the corrective action includes providing a warning via a graphical interface.
8. The method of claim 7 , further comprising:
performing a second corrective action for the predictive model when the indication of the predictive ability decay reaches a second threshold, wherein the second corrective action includes issuing an alert via the graphical interface.
9. The method of claim 8 , wherein the threshold is indicative of less severe predictive decay than the second threshold.
10. The method of claim 1 , wherein comparing further includes identifying a change in an input or output variable distribution, and wherein the identified change is greater than a distribution change threshold.
11. The method of claim 1 , wherein the corrective action includes building a replacement predictive model.
12. The method of claim 11 , wherein the replacement predictive model is automatically recalibrated using the new data.
13. The method of claim 1 , wherein the corrective action includes sending a notification to pre-selected parties.
14. The method of claim 1 , further comprising:
logging the baseline performance metric and the updated performance metric in a data store.
15. The method of claim 14 , further comprising:
providing monitoring reports showing trends in model performance.
16. The method of claim 1 , wherein the predictive model is retrieved from a centralized model repository.
17. The method of claim 1 , wherein the indication of predictive ability decay is calculated according to a Lift 5 decay, wherein the Lift 5 decay identifies a lift performance decay based upon top 5% of the data and the new data, wherein the Lift 5 decay is calculated according to:
Lift5decay=(Lift5 B −Lift5 A )/Lift5 A ,
where Lift 5 A is the lift value of the predictions of the predictive model for the top 5% based on the data, where Lift 5 B is the lift value of the new predictions of the predictive model for the top 5% based on the new data.
18. The method of claim 1 , wherein the indication of predictive ability decay is calculated according to a Lift 10 decay, wherein the Lift 10 decay identifies a lift performance decay based on a top 10% of the data and the new data, wherein the Lift 10 decay is calculated according to:
Lift10decay=(Lift10 B −Lift10 A )/Lift10 A ,
where Lift 10 A is the lift value of the predictions of the predictive model for the top 10% based on the data, where Lift 10 B is the lift value of the new predictions of the predictive model for the top 10% based on the new data.
19. The method of claim 1 , wherein the indication of predictive ability decay is calculated according to a GiniDecay, wherein the GiniDecay identifies a decline in an area under a receiver operating curve, wherein the GiniDecay is calculated according to:
GiniDecay=(GiniIndex B −GiniIndex A )/GiniIndex A ,
where GiniIndex A is the area under the receiver operating curve for the predictions of the predictive model based on the data, where GiniIndex B is the area under the receiver operating curve for the new predictions of the predictive model based on the new data.
20. The method of claim 1 , wherein the indication of predictive ability decay is calculated according to a KSDecay, wherein the KSDecay identifies a decline in a maximum vertical distance between two curves of a Kolmogorov-Smirnov chart, wherein the KSDecay is calculated according to:
KSDecay=(KSStatistic B −KSStatistic A )/KSStatistic A ,
where KSStatistic A is the vertical distance between the two curves for the predictions of the predictive model based on the data, where KSStatistic B is the vertical distance between the two curves for the new predictions of the predictive model based on the new data.
21. The method of claim 17 , wherein the data and the new data include actual values.
22. A system for assessing a predictive model, comprising:
one or more data processors;
one or more computer-readable storage mediums containing instructions configured to cause the one or more processors to perform operations including:
accessing a predictive model for scoring transactional data and data associated with a time period;
generating one or more predictions using the predictive model and the data;
calculating a baseline performance metric for the predictive model based upon the predictions;
accessing new data associated with a new time period that is later than the time period;
generating one or more new data predictions using the predictive model and the new data;
calculating an updated performance metric for the predictive model based upon the new data predictions;
comparing the baseline performance metric and the updated performance metric to determine an indication of predictive ability decay for the predictive model; and
performing a corrective action for the predictive model when the indication of predictive ability decay reaches a threshold.
23. A computer-program product for assessing a predictive model, tangibly embodied in a machine-readable storage medium, including instructions configured to cause a data processing apparatus to:
access a predictive model for scoring transactional data and data associated with a time period;
generate one or more predictions using the predictive model and the data;
calculate a baseline performance metric for the predictive model based upon the predictions;
access new data associated with a new time period that is later than the time period;
generate one or more new data predictions using the predictive model and the new data;
calculate an updated performance metric for the predictive model based upon the new data predictions;
compare the baseline performance metric and the updated performance metric to determine an indication of predictive ability decay for the predictive model; and
perform a corrective action for the predictive model when the indication of predictive ability decay reaches a threshold.