System and method of scaling-aware rating of count forecasts
A system and method are disclosed for evaluating a forecast. The method includes aggregating a set of predictions and a set of actuals into prediction buckets, rating prediction data of the set of predictions using forecast metrics, computing an ideal benchmark for the forecast metrics, determining a scaling adjustment to calculate one or more non-ideal benchmarks to create a forecast rating scale, determining a score for each of the prediction buckets, the score corresponding to the forecast rating scale; and aggregating the determined scores of the prediction buckets to generate an overall score. The method further includes generating a calibration diagram comprising observed frequency and predicted probability of the set of predictions and the set of actuals.
1 . A computer-implemented method for evaluating a forecast, comprising:
aggregating, by a computer comprising a processor and memory, a set of predictions and a set of actuals into one or more prediction buckets, wherein the aggregating separates high volume and low volume products and accounts for a shelf-life of perishable goods;
rating, by the computer, prediction data of the set of predictions using one or more forecast metrics;
computing, by the computer, an ideal benchmark for the one or more forecast metrics;
determining, by the computer, a scaling adjustment to calculate one or more non-ideal benchmarks to create a forecast rating scale;
determining, by the computer, a score for each of the one or more prediction buckets, the score corresponding to the forecast rating scale;
aggregating, by the computer, the determined one or more scores of the one or more prediction buckets to generate an overall score; and
improving, by the computer, a probability density function by iteratively applying an update rule, wherein a probability density that contributes to observations that are underpredicted is boosted.
2 . The computer-implemented method of claim 1 , wherein the one or more prediction buckets are logarithmically spaced.
3 . The computer-implemented method of claim 1 , further comprising:
determining, by the computer, the scaling adjustment by simulating non-Poissonian dispersions.
4 . The computer-implemented method of claim 1 , wherein the one or more forecast metrics comprise one or more functions that produce a numerical rating that compares one or more prediction values with one or more actual values.
5 . The computer-implemented method of claim 1 , further comprising:
associating, by the computer, a percentage scale with the forecast rating scale.
6 . The computer-implemented method of claim 1 , further comprising:
depicting, by the computer, the overall score as a forecast rating graph.
7 . The computer-implemented method of claim 1 , further comprising:
generating, by the computer, a calibration diagram comprising observed frequency and predicted probability of the set of predictions and the set of actuals.
8 . A system for evaluating a forecast, comprising:
a computer, comprising a processor and memory, the computer configured to:
aggregate a set of predictions and a set of actuals into one or more prediction buckets, wherein the aggregating separates high volume and low volume products and accounts for a shelf-life of perishable goods;
rate prediction data of the set of predictions using one or more forecast metrics;
compute an ideal benchmark for the one or more forecast metrics;
determine a scaling adjustment to calculate one or more non-ideal benchmarks to create a forecast rating scale;
determine a score for each of the one or more prediction buckets, the score corresponding to the forecast rating scale;
aggregate the determined one or more scores of the one or more prediction buckets to generate an overall score; and
improve a probability density function by iteratively applying an update rule, wherein a probability density that contributes to observations that are underpredicted is boosted.
9 . The system of claim 8 , wherein the one or more prediction buckets are logarithmically spaced.
10 . The system of claim 8 , wherein the computer is further configured to:
determine the scaling adjustment by simulating non-Poissonian dispersions.
11 . The system of claim 8 , wherein the one or more forecast metrics comprise one or more functions that produce a numerical rating that compares one or more prediction values with one or more actual values.
12 . The system of claim 8 , wherein the computer is further configured to:
associate a percentage scale with the forecast rating scale.
13 . The system of claim 8 , wherein the computer is further configured to:
depict the overall score as a forecast rating graph.
14 . The system of claim 8 , wherein the computer is further configured to:
generate a calibration diagram comprising observed frequency and predicted probability of the set of predictions and the set of actuals.
15 . A non-transitory computer-readable medium embodied with software for evaluating a forecast, the software when executed:
aggregates a set of predictions and a set of actuals into one or more prediction buckets, wherein the aggregating separates high volume and low volume products and accounts for a shelf-life of perishable goods;
rates prediction data of the set of predictions using one or more forecast metrics;
computes an ideal benchmark for the one or more forecast metrics;
determines a scaling adjustment to calculate one or more non-ideal benchmarks to create a forecast rating scale;
determines a score for each of the one or more prediction buckets, the score corresponding to the forecast rating scale;
aggregates the determined one or more scores of the one or more prediction buckets to generate an overall score; and
improves a probability density function by iteratively applying an update rule, wherein a probability density that contributes to observations that are underpredicted is boosted.
16 . The non-transitory computer-readable medium of claim 15 , wherein the one or more prediction buckets are logarithmically spaced.
17 . The non-transitory computer-readable medium of claim 15 , wherein the software when executed further:
determines the scaling adjustment by simulating non-Poissonian dispersions.
18 . The non-transitory computer-readable medium of claim 15 , wherein the one or more forecast metrics comprise one or more functions that produce a numerical rating that compares one or more prediction values with one or more actual values.
19 . The non-transitory computer-readable medium of claim 15 , wherein the software when executed further:
associates a percentage scale with the forecast rating scale.
20 . The non-transitory computer-readable medium of claim 15 , wherein the software when executed further:
depicts the overall score as a forecast rating graph.