Nonparametric method for determination of anomalous event states in complex systems exhibiting non-stationarity

1. A method comprising:

1. initializing a system for determining anomalous event states in an information technology (IT) infrastructure by:

(a) collecting historical time-series data for a metric in the IT infrastructure for at least one monitoring period, each monitoring period having a plurality of timeslots and each timeslot observing metric data at least once;

(b) screening the historical data to remove abnormal metric data to form a non-abnormal historical data set;

(c) collecting each data point from the non-abnormal historical data set into a timeslot historical data set that corresponds with the timeslot from which each data point was collected historically;

(d) simulating m monitoring periods by:

(i) generating a simulated time-series data stream comprising randomly selected data points selected from the timeslot historical data set corresponding to the same timeslot in the simulated time-series data stream;

(ii) calculating a cumulative sum value for each simulated randomly selected data point; and

(iii) determining a maximum cusum value (max {S n } m ) for the simulated monitoring period, where n is an indicator of sequential position of each data point in the data stream

to create a set {max{S n } 1 , max{S n } 2 ,...,max{S n } m }; and

(e) selecting as a threshold max{max{S n } 1 , max{S n } 2 ,...,max{S n } m };

2. comparing incoming data for each metric against the threshold by

performing a cumulative sum analysis on a set of incoming metric data in real time by calculating a cumulative sum value for each metric and comparing the cumulative sum value to the threshold to detect anomalous events: and

3. initiating an alert state when an anomalous event is detected that effects at least one response to address the conditions triggering the anomalous event in the affected component of the IT infrastructure:

wherein each cumulative sum value is calculated by adding to the previous cumulative sum value the difference between the value of a metric and a value for normal behavior for a current timeslot within the monitoring period: and

wherein if the calculation of the cumulative sum value is negative, it is set to zero.

2. The method of claim 1 , further comprising reinitializing after the completion of each monitoring period.

3. The method of claim 2 , further comprising:

storing an initialization data set after each initialization;

during the reinitialization process, removing the oldest monitoring period of initialization data from the initialization data set and adding the immediately previous monitoring period's data to the initialization data set to form an updated initialization data set;

wherein the reinitialization process uses the updated initialization data set.

4. The method of claim 1 , further comprising:

determining the end point of each anomalous event.

5. The method of claim 4 , wherein the determination of the end point of each anomalous event is designated as the largest cusum value in the set of ν historical cusum values having negative or flat regression slope;

wherein regressions over ν cusum values begins at the first cusum statistic to exceed the threshold and progresses for each cusum statistic thereafter until a negative or flat slope is calculated.

6. The method of claim 1 , further comprising:

determining a start point of each anomalous event.

7. The method of claim 6 , wherein the determination of the start point of each anomalous event is designated as the most recent cumulative sum value in the set of νcumulative sum values {S n ,S n−1 ,...,S n−ν+1 } having the first negative or flat regression slope;

wherein S n is initially the first cumulative sum value in an anomalous event to exceed the threshold and regressions over νpoints are calculated until a negative or flat slope results, each calculation taking the regression for the next n−1 set of cusum values.

8. The method of claim 1 , wherein the value for normal behavior is a function of a tunable parameter that determines whether a given metric is outside of a range of normal values for the timeslot in which the given metric is observed.

9. The method of claim 8 , wherein at least one cumulative sum value is calculated by the function:

S n =max{0,S n−1 +Y n −Q τ n (α)}

wherein n represents a timeslot, S n is the cumulative sum value, Y n is the incoming metric value, Q τ n (α) is a function defining an extreme value for normal behavior based on historical metric values for the same timeslot in which Y n is observed and tunable parameter α,and S 0 =0.

10. The method of claim 8 , wherein at least two cumulative sum values are calculated by the functions:

S n 30 =max{0,S n− 30 +Y n −Q τ n(α)}

S n 31 =max{0,S n−1 31 +Q τ n (1−α)−Y n}

wherein n represents a timeslot; S n + , S n − are cumulative sum values; Y n is the incoming metric value; Q τ n (α) is a function defining a maximum value for normal behavior based on historical metric values for the same timeslot in which Y n is observed and tunable parameter α; Q τ n (1−α) is a function defining a minimum lower value for normal behavior based on historical metric values for the same timeslot in which Y n is observed and tunable parameter α; and S 0 + =0, S 0 − =0.

11. A tangible machine-readable medium having program instructions stored thereon executable by a processing unit for performing the steps of:

1. initializing a system for determining anomalous event states in an information technology (IT) infrastructure by:

(a)collecting historical time-series data for a metric in the IT infrastructure for at least one monitoring period, each monitoring period having a plurality of timeslots and each timeslot observing metric data at least once;

(b) screening the historical data to remove abnormal metric data to form a non-abnormal historical data set;

(c) collecting each data point from the non-abnormal historical data set into a timeslot historical data set that corresponds with the timeslot from which each data point was collected historically;

(d) simulating m monitoring periods by:

(i) generating a simulated time-series data stream comprising randomly selected data points selected from the timeslot historical data set corresponding to the same timeslot in the simulated time-series data stream;

(ii) calculating a cumulative sum value for each simulated randomly selected data point; and

(iii) determining a maximum cusum value (max{S n } m ) for the simulated monitoring period, where n is an indicator of sequential position of each data point in the data stream

to create a set {max{S n } 1 , max{S n } 2 , . . . , max{S n } m }; and

(e) selecting as a threshold max{max{S n } 1 ,max{S n } 2 ,..., max{S n } m };

2. comparing incoming data for each metric against the threshold by

performing a cumulative sum analysis on a set of incoming metric data in real time by calculating a cumulative sum value for each metric and comparing the cumulative sum value to the threshold to detect anomalous events: and

3. initiating an alert state when an anomalous event is detected that effects at least one response to address the conditions triggering the anomalous event in the affected component of the IT infrastructure:

wherein each cumulative sum value is calculated by adding to the previous cumulative sum value the difference between the value of a metric and a value for normal behavior for a current timeslot within the monitoring period: and

wherein if the calculation of the cumulative sum value is negative, it is set to zero.

12. The tangible machine-readable medium of claim 11 , further comprising reinitializing after the completion of each monitoring period.

13. The tangible machine-readable medium of claim 12 , further comprising:

storing an initialization data set after each initialization;

during the reinitialization process, removing the oldest monitoring period of initialization data from the initialization data set and adding the immediately previous monitoring period's data to the initialization data set to form an updated initialization data set;

wherein the reinitialization process uses the updated initialization data set.

14. The tangible machine-readable medium of claim 12 , further comprising:

determining the end point of each anomalous event.

15. The tangible machine-readable medium of claim 14 , wherein the determination of the end point of each anomalous event is designated as the largest cusum value in the set of νhistorical cusum values having negative or flat regression slope;

wherein regressions over νcusum values begins at the first cusum statistic to exceed the threshold and progresses for each cusum statistic thereafter until a negative or flat slope is calculated.

16. The tangible machine-readable medium of claim 11 , further comprising:

determining a start point of each anomalous event.

17. The tangible machine-readable medium of claim 16 , wherein the determination of the start point of each anomalous event is designated as the most recent cumulative sum value in the set of νcumulative sum values {S n ,S n− , . . . ,S n−ν+1 } having the first negative or flat regression slope;

wherein S n is initially the first cumulative sum value in an anomalous event to exceed the threshold and regressions over ν points are calculated until a negative or flat slope results, each calculation taking the regression for the next n− 1 set of cusum values.

18. The tangible machine-readable medium of claim 11 , wherein the value for normal behavior is a function of a tunable parameter that determines whether a given metric is outside of a range of normal values for the timeslot in which the given metric is observed.

19. The tangible machine-readable medium of claim 18 , wherein at least one cumulative sum value is calculated by the function:

S n =max{0,S n−1 +Y n −Qτ n (α)}

wherein n represents a timeslot, S, n is the cumulative sum value, Y n is the incoming metric value, Q τ n (α) is a function defining an extreme value for normal behavior based on historical metric values for the same timeslot in which Y n is observed and tunable parameter α,and S 0 =0.

20. The tangible machine-readable medium of claim 18 , wherein at least two cumulative sum values are calculated by the functions:

S n + =max{0,S n−1 + +Y n −Q τ n (α)}

S n − =max{0,S n−1 − +Q τ n (1α)−Y n }

wherein n represents a timeslot; S n +, S n 31 are cumulative sum values; Y n is the incoming metric value; Q τ n (α) is a function defining a maximum value for normal behavior based on historical metric values for the same timeslot in which Y n is observed and tunable parameter α; Q τ n (1−α) is a function defining a minimum value for normal behavior based on historical metric values for the same timeslot in which Y n is observed and tunable parameter α; and S 0 + =0, S 0 − =0.