Processing data streams received from instrumented software in real time using incremental-decremental implementation of the KPSS stationarity statistic
An analysis system receives a time series. The data values of the time series correspond to a metric describing a characteristic of the computing system that changes over time. The analysis system stores a statistic value that represents the stationarity of the time series. In response to receiving a most recent value, the analysis system assigns the most recent value as the leading value in a window before retrieving the trailing value of the window. The analysis system updates the statistic value to add an influence of the most recent value and remove an influence of the trailing value. If the statistic value is less than a threshold, the analysis system determines that the time series is stationary. In response to determining the time series is stationary, the analysis system assigns an alert to the metric. The analysis system detects an anomaly in the metric based on the assigned alert.
1. A method for detecting an anomaly in a metric for a computing system, the method comprising:
receiving a most recent value of a plurality of data values in a time series, the time series comprising at least a plurality of data values, each of the plurality of data values corresponding to a metric;
assigning the most recent value as a leading value in a window, the window comprising at least a leading value and a trailing value;
updating a statistic value to add an influence of the most recent value and remove an influence of the trailing value, the statistic value based at least in part on normalized partial sums of residual values, the statistic value comprising a normalized series of partial sums of residual values in a series of residual values;
determining whether the time series is stationary by determining whether the statistic value is less than a first threshold;
in response to determining that the time series is stationary, assigning an alert for a stationary data stream to the metric; and
detecting an anomaly in the metric based on the assigned alert.
2. The method of claim 1 , further comprising:
storing a statistic value representing a current stationarity of the time series based on data values of the plurality of data values for the time series in a window.
3. The method of claim 1 , further comprising:
retrieving the trailing value of the window.
4. The method of claim 1 , further comprising:
storing a plurality of tracked values calculated based on data values in the window, the plurality of tracked values comprising at least one of a count value, a sum value, a sum squares value, a weighted sum value, a sum sizes value, and/or a sum squared size value.
5. The method of claim 1 , wherein the residual values comprise a difference of a mean and each data value of the time series, and each partial sum of the series of partial sums is a summation of a corresponding residual value and residual values that occur before the corresponding residual value in the series of residual values.
6. The method of claim 1 , further comprising:
updating a plurality of tracked values to add an influence of the most recent value and remove an influence of the trailing value, wherein updating the statistic value to add an influence of the most recent value and remove an influence of the trailing value comprises calculating the statistic value based on the plurality of tracked values.
7. The method of claim 1 , wherein the statistic value being greater than a second threshold indicates that the time series is not stationary, and the statistic value being greater than the first threshold but less than the second threshold indicates that a stationarity of the time series is ambiguous.
8. The method of claim 1 , wherein the metric comprises a characteristic of a computing system that changes over time.
9. A system for detecting an anomaly in a metric for a computing system, the system comprising:
at least one memory having instructions stored thereon; and
at least one processor configured to execute the instructions, wherein the at least one processor is configured to:
receive a most recent value of a plurality of data values in a time series, the time series comprising at least a plurality of data values, each of the plurality of data values corresponding to a metric;
assign the most recent value as a leading value in a window, the window comprising at least a leading value and a trailing value;
update a statistic value to add an influence of the most recent value and remove an influence of the trailing value, the statistic value based at least in part on normalized partial sums of residual values, the statistic value comprising a normalized series of partial sums of residual values in a series of residual values;
determine whether the time series is stationary by determining whether the statistic value is less than a first threshold;
in response to determining that the time series is stationary, assign an alert for a stationary data stream to the metric; and
detect an anomaly in the metric based on the assigned alert.
10. The system of claim 9 , further configured to:
store a statistic value representing a current stationarity of the time series based on data values of the plurality of data values for the time series in a window.
11. The system of claim 9 , further configured to:
retrieve the trailing value of the window.
12. The system of claim 9 , further configured to:
store a plurality of tracked values calculated based on data values in the window, the plurality of tracked values comprising at least one of a count value, a sum value, a sum squares value, a weighted sum value, a sum sizes value, and/or a sum squared size value.
13. The system of claim 9 , wherein the residual values comprise a difference of a mean and each data value of the time series, and each partial sum of the series of partial sums is a summation of a corresponding residual value and residual values that occur before the corresponding residual value in the series of residual values.
14. The system of claim 9 , further configured to:
update a plurality of tracked values to add an influence of the most recent value and remove an influence of the trailing value, wherein updating the statistic value to add an influence of the most recent value and remove an influence of the trailing value comprises calculating the statistic value based on the plurality of tracked values.
15. The system of claim 9 , wherein the statistic value being greater than a second threshold indicates that the time series is not stationary, and the statistic value being greater than the first threshold but less than the second threshold indicates that a stationarity of the time series is ambiguous.
16. The system of claim 9 , wherein the metric comprises a characteristic of a computing system that changes over time.
17. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform operations for processing data generated by instrumented software, comprising:
receiving a most recent value of a plurality of data values in a time series, the time series comprising at least a plurality of data values, each of the plurality of data values corresponding to a metric;
assigning the most recent value as a leading value in a window, the window comprising at least a leading value and a trailing value;
updating a statistic value to add an influence of the most recent value and remove an influence of the trailing value, the statistic value based at least in part on normalized partial sums of residual values, the statistic value comprising a normalized series of partial sums of residual values in a series of residual values;
determining whether the time series is stationary by determining whether the statistic value is less than a first threshold;
in response to determining that the time series is stationary, assigning an alert for a stationary data stream to the metric; and
detecting an anomaly in the metric based on the assigned alert.
18. The non-transitory computer-readable storage medium of claim 17 , further configured for:
storing a statistic value representing a current stationarity of the time series based on data values of the plurality of data values for the time series in a window.