Forecast model quality index for computer storage capacity planning
View Patent ↗A robust, simple, application-specific way to evaluate data models and forecasts is provided for evaluating whether a forecast is trustworthy. An approach for formulating a single, summary indication of data/model/forecast quality relevant for the task at hand is described. This approach includes generating a forecast model from collected data, combining multiple model-quality parameters based on the model, computing an indication based on the combination of parameters, and evaluating the model and forecast quality based on the indication. This indication, in the form of a Model Quality Index, can also be used to compare different types of models produced by different types of analysis approaches.
1. A system for integrating the computation for statistical modeling and forecasting, the system comprising:
a computer system comprising computer hardware, the computer system programmed to implement:
a data pattern analysis engine configured to:
receive collected data related to computer storage capacity,
detect any trend in the collected data, and
detect any seasonality in the data;
a forecast engine configured to select a forecast model adapted to predict an amount of additional computer storage capacity to obtain in the future and when to obtain the additional computer storage capacity, the forecast model operative to select the forecast model based at least partly on any said outliers found, any said trend detected, and any said seasonality;
a model quality engine configured to:
calculate a model quality index, the model quality index reflecting a quality of the forecast model, wherein said calculation by the model quality engine comprises:
calculating a plurality of model-quality indicators by at least analyzing the forecast model, said model-quality indicators comprising two or more of the following: an R 2 goodness of fit, a Theil's U score, a Ljung-Box score, a missed seasonality score, a data quality score, a forecast horizon score, and a sample size, and
combining the plurality of model-quality indicators into a weighted sum to produce the model quality index; and
adjust one or more parameters of the forecast model based at least in part on a value of the model quality index, said one or more parameters of the forecast model comprising a confidence level, a user preference, a forecast horizon confidence, a critical value, and one or more weight factors.
2. The system of claim 1 , wherein the model quality engine is further configured to compare the model quality index to a threshold.
3. The system of claim 2 , wherein the model quality engine is further configured to adjust the one or more parameters of the forecast model responsive to said comparison.
4. A method for integrating the computation for statistical modeling and forecasting, the method comprising:
by a computer system comprising computer hardware:
generating a forecast model from collected data, the forecast model adapted to predict an amount of additional computer storage capacity to obtain in the future and when to obtain the additional computer storage capacity, said generating the forecast model comprising:
finding and interpolating any outliers in the collected data,
detecting any trend in the collected data,
detecting any seasonality in the data, and
generating the forecast model based at least partly on any said outliers found, any said trend detected, and any said seasonality detected;
calculating a model quality index, the model quality index reflecting a quality of the forecast model, said calculating comprising:
calculating a plurality of model-quality indicators by at least analyzing the forecast model, said model-quality indicators comprising a goodness of fit, a missed seasonality score, and a sample size, and
combining the plurality of model-quality indicators into a weighted sum to produce the model quality index; and
adjusting one or more parameters of the forecast model to predict storage capacity based at least in part on a value of the model quality index, said one or more parameters of the forecast model comprising a confidence level, a user preference, a forecast horizon confidence, a critical value, and one or more weight factors.
5. The method of claim 4 , further comprising comparing the model quality index to a threshold.
6. The method of claim 4 , wherein said adjusting is performed responsive to said comparison.
7. The method of claim 4 , wherein weights of the weighted sum are tuned to different transitional values.
8. The method of claim 4 , wherein weights of the weighted sum are user-adjustable.
9. The method of claim 4 , further comprising outputting an indication to the user of model quality, the indication reflecting the model quality index.
10. The method of claim 9 , wherein said indication is quantized to a level selected from the group consisting of poor, good, and excellent.
11. The method of claim 4 , wherein said adjusting comprises tuning the model by considering one or more parameters of the forecast model selected by a user.
12. A computer-readable medium having computer program instructions configured to implement a method of statistical modeling and forecasting, method comprising:
by a computer system comprising computer hardware:
generating a forecast model from collected data related to computer storage capacity, the forecast model adapted to predict an amount of additional computer storage capacity to obtain in the future and when to obtain the additional computer storage capacity;
calculating a model quality index, the model quality index reflecting a quality of the forecast model, said calculating comprising calculating a plurality of model-quality indicators, said model-quality indicators comprising two or more of the following: an R 2 goodness of fit, a Theil's U score, a Ljung-Box score, a missed seasonality score, a data quality score, a forecast horizon score, and a sample size; and
adjusting one or more parameters of the forecast model to predict storage capacity based at least in part on the model quality index, said one or more parameters of the forecast model comprising one or more of the following: a confidence level, a user preference, a forecast horizon confidence, and a critical value.
13. The computer-readable medium of claim 12 , further comprising comparing the model quality index to a threshold.
14. The computer-readable medium of claim 12 , wherein said adjusting is performed responsive to said comparison.
15. The computer-readable medium of claim 12 , wherein calculating the model quality index comprises calculating a weighted combination of one or more of the model-quality indicators.
16. The computer-readable medium of claim 15 , wherein weights of the weighted combination are tuned to different transitional values.
17. The computer-readable medium of claim 15 , wherein weights of the weighted combination are user-adjustable.
18. The computer-readable medium of claim 12 , further comprising outputting an indication to the user of model quality, the indication reflecting the model quality index.
19. The computer-readable medium of claim 18 , wherein said indication is quantized to a level selected from the group consisting of poor, good, and excellent.
20. The computer-readable medium of claim 12 , wherein said adjusting comprises tuning the model by considering one or more parameters of the forecast model selected by a user.
21. The method of claim 4 , wherein said model quality indicators further comprise one or more of the following: a Theil's U score, a Ljung-Box score, a data quality score, and a forecast horizon score.
22. The computer program product of claim 12 , wherein the forecast model is generated by at least:
finding and interpolating any outliers in the collected data;
detecting any trend in the collected data; and
detecting any seasonality in the data.