Automatic determination of high significance alert thresholds for system performance metrics using an exponentially tailed model
A computer is programmed to fit exponential models to upper percentile subsets of observed measurements for performance metrics collected as attributes of a computer system. The subsets are defined from sets chosen to reduce model bias due to expected variations in system performance, e.g. those resulting from temporal usage patterns induced by end users and/or workload scheduling. Measurement levels corresponding to high cumulative probability, indicative of likely performance anomalies, are extrapolated from the fitted models generated from measurements of lower cumulative probability. These levels are used to establish and to automatically set warning and alert thresholds which signal to (human) administrators when performance anomalies are observed.
1. A computer implemented method of determining a threshold for a metric, the method comprising:
partitioning a plurality of measurements of the metric into a number of sets including a set, the set comprising measurements collected during a plurality of intervals of time;
selecting a subset of measurements from the set whose rank occurs within a predetermined percentile range;
fitting measurements in the subset to a statistical distribution function having a tail that decays slower than a normal distribution, to obtain at least two parameters thereof;
computing, based on said at least two parameters, a value of the metric at a predetermined probability outside of the predetermined percentile range; and
performing an action when a new measurement of the metric, in a new interval of time corresponding to the set, crosses said value obtained from the computing.
2. The method of claim 1 wherein:
a lower limit of the predetermined percentile range is greater than or equal to 90%; and
an upper limit of the predetermined percentile range is less than the predetermined probability.
3. The method of claim 1 wherein:
each interval in the plurality of intervals is separated from another interval in the plurality of intervals, by a common predetermined duration.
4. The method of claim 1 further comprising:
displaying to an operator a plurality of schemes for partitioning measurements based on time; and
receiving, from the operator, a scheme selected from the plurality of schemes.
5. The method of claim 4 wherein the plurality of schemes comprise at least one of:
a daily partition that divides up measurements made during each day; and
a weekly partition that divides up measurements made during each week.
6. The method of claim 5 wherein the daily partition is one of:
each hour of the day;
day and night; and
entire day.
7. The method of claim 5 wherein the weekly partition is one of:
each day of the week;
weekday and weekend; and
entire week.
entire week.
8. The method of claim 5 wherein at least one scheme in the plurality of schemes is related to a periodicity induced by human usage of a system.
9. The method of claim 5 wherein at least one scheme in the plurality of schemes is related to a periodicity induced by scheduled jobs of a system.
10. A computer-readable storage medium encoded with instructions to determine a threshold for a metric, the instructions comprising:
partitioning a plurality of measurements of the metric into a number of sets including a set, the set comprising measurements collected during a plurality of intervals of time;
selecting a subset of measurements from the set whose rank occurs within a predetermined percentile range:
fitting measurements in the subset to a statistical distribution function having a tail that decays slower than a normal distribution, to obtain at least two parameters thereof;
computing, based on said at least two parameters, a value of the metric at a predetermined probability outside of the predetermined percentile range; and
performing an action when a new measurement of the metric, in a new interval of time corresponding to the set, crosses said value obtained from the computing.
11. The computer-readable storage medium of claim 10 wherein:
a lower limit of the predetermined percentile range is greater than or equal to 90%; and
an upper limit of the predetermined percentile range is less than the predetermined probability.
12. The computer-readable storage medium of claim 10 wherein:
each interval in the plurality of intervals is separated from another interval in the plurality of intervals, by a common predetermined duration.
13. The computer-readable storage medium of claim 10 wherein said instructions further comprise:
displaying to an operator a plurality of schemes for partitioning measurements based on time; and
receiving, from the operator, a scheme selected from the plurality of schemes.
14. The computer-readable storage medium of claim 13 wherein the plurality of schemes comprise at least one of:
a daily partition that divides up measurements made during each day; and
a weekly partition that divides up measurements made during each week.
15. The computer-readable storage medium of claim 13 wherein at least one scheme in the plurality of schemes is related to a periodicity induced by human usage of a system.
16. The computer-readable storage medium of claim 13 wherein at least one scheme in the plurality of schemes is related to a periodicity induced by scheduled jobs of a system.