Processing data streams received from instrumented software in real time using incremental-decremental implementation of the KPSS stationarity statistic
An analysis system receives a time series. The data values of the time series correspond to a metric describing a characteristic of the computing system that changes over time. The analysis system stores a statistic value that represents the stationarity of the time series. In response to receiving a most recent value, the analysis system assigns the most recent value as the leading value in a window before retrieving the trailing value of the window. The analysis system updates the statistic value to add an influence of the most recent value and remove an influence of the trailing value. If the statistic value is less than a threshold, the analysis system determines that the time series is stationary. In response to determining the time series is stationary, the analysis system assigns an alert to the metric. The analysis system detects an anomaly in the metric based on the assigned alert.
1. A computer-implemented method for detecting an anomaly in a metric for a computing system, the method comprising:
receiving a most recent value of a plurality of data values in a time series comprising multiple data values corresponding to multiple metrics;
determining that the most recent value comprises a leading value or a trailing value in a window;
updating a statistic value to add an influence of the most recent value or remove an influence of the trailing value;
determining whether the statistic value is less than a first threshold, wherein the statistic value being greater than a second threshold indicates that the time series is not stationary, and the statistic value being greater than the first threshold but less than the second threshold indicates that a stationarity of the time series is ambiguous;
in response to the determining that the statistic value is less than the first threshold, assigning alerts for stationary data streams of the metrics; and
determining anomalies in the metrics based on the assigned alerts.
2. The computer-implemented method of claim 1 , further comprising:
storing a statistic value representing a current stationarity of the time series based on data values of the plurality of data values for the time series in a window.
3. The computer-implemented method of claim 1 , further comprising:
retrieving the trailing value of the window.
4. The computer-implemented method of claim 1 , further comprising:
storing a plurality of tracked values calculated based on data values in the window, the plurality of tracked values comprising at least one of a count value, a sum value, a sum squares value, a weighted sum value, a sum sizes value, and/or a sum squared size value.
5. The computer-implemented method of claim 1 , wherein the statistic value comprises a normalized series of partial sums of residual values in a series of residual values, wherein residual values comprise a difference of a mean and each data value of the time series, and each partial sum of a series of partial sums is a summation of a corresponding residual value and residual values that occur before the corresponding residual value in the series of residual values.
6. The computer-implemented method of claim 1 , further comprising:
updating a plurality of tracked values to add an influence of the most recent value and remove an influence of the trailing value, wherein updating the statistic value to add an influence of the most recent value and remove an influence of the trailing value comprises calculating the statistic value based on the plurality of tracked values.
7. The computer-implemented method of claim 1 , wherein the metric comprises a characteristic of a computing system that changes over time.
8. A system for detecting an anomaly in a metric for a computing system, the system comprising:
at least one memory having instructions stored thereon; and
at least one processor configured to execute the instructions, wherein the at least one processor is configured to:
receive a most recent value of a plurality of data values in a time series comprising multiple data values corresponding to multiple metrics;
determine that the most recent value comprises a leading value or a trailing value in a window;
update a statistic value to add an influence of the most recent value or remove an influence of the trailing value;
determine whether the statistic value is less than a first threshold, wherein the statistic value being greater than a second threshold indicates that the time series is not stationary, and the statistic value being greater than the first threshold but less than the second threshold indicates that a stationarity of the time series is ambiguous;
in response to determining that the statistic value is less than the first threshold, assigning alerts for stationary data streams of the metrics; and
determine anomalies in the metrics based on the assigned alerts.
9. The system of claim 8 , further configured to:
store a statistic value representing a current stationarity of the time series based on data values of the plurality of data values for the time series in a window.
10. The system of claim 8 , further configured to:
retrieve the trailing value of the window.
11. The system of claim 8 , further configured to:
store a plurality of tracked values calculated based on data values in the window, the plurality of tracked values comprising at least one of a count value, a sum value, a sum squares value, a weighted sum value, a sum sizes value, and/or a sum squared size value.
12. The system of claim 8 , wherein the statistic value comprises a normalized series of partial sums of residual values in a series of residual values, wherein residual values comprise a difference of a mean and each data value of the time series, and each partial sum of a series of partial sums is a summation of a corresponding residual value and residual values that occur before the corresponding residual value in the series of residual values.
13. The system of claim 8 , further configured to:
update a plurality of tracked values to add an influence of the most recent value and remove an influence of the trailing value, wherein updating the statistic value to add an influence of the most recent value and remove an influence of the trailing value comprises calculating the statistic value based on the plurality of tracked values.
14. The system of claim 8 , wherein the metric comprises a characteristic of a computing system that changes over time.
15. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform operations for processing data generated by instrumented software, comprising:
receiving a most recent value of a plurality of data values in a time series comprising multiple data values corresponding to multiple metrics;
determining that the most recent value comprises a leading value or a trailing value in a window;
updating a statistic value to add an influence of the most recent value or remove an influence of the trailing value;
determining whether the statistic value is less than a first threshold, wherein the statistic value being greater than a second threshold indicates that the time series is not stationary, and the statistic value being greater than the first threshold but less than the second threshold indicates that a stationarity of the time series is ambiguous;
in response to the determining that the statistic value is less than the first threshold, assigning alerts for stationary data streams of the metrics; and
determining anomalies in the metrics based on the assigned alerts.
16. The non-transitory computer-readable storage medium of claim 15 , further configured for:
storing a statistic value representing a current stationarity of the time series based on data values of the plurality of data values for the time series in a window.
17. The non-transitory computer-readable storage medium of claim 15 , further configured for:
retrieving the trailing value of the window.
18. The non-transitory computer-readable storage medium of claim 15 , further configured for:
storing a plurality of tracked values calculated based on data values in the window, the plurality of tracked values comprising at least one of a count value, a sum value, a sum squares value, a weighted sum value, a sum sizes value, and/or a sum squared size value.