Detecting anomalies in a time series
View Patent ↗Embodiments are disclosed for detecting anomalies in time series using statistical models. In some embodiments, a linear regression model is built for the time series for predicting future values of the time series. Furthermore, the standard deviation of the difference between a prediction and an ideal value of the time series at any point is then estimated. An anomaly is detected when the difference between the prediction and the observed value is greater than a certain threshold based on the estimated standard deviation.
1. A computer-implemented method of detecting anomalies in time series data, comprising:
modeling, by a computing device having a processor and a memory, a time series using a linear regression framework;
representing, by the processor, the time series as a sum of a signal portion and a noise, wherein the signal portion corresponds to time-dependent data, and the noise removes time dependence from the data;
calculating, by the processor, for a specified time, a variance of a difference between a value of the time series predicted by the linear regression framework and a value of the signal portion;
determining, by the processor, a threshold based on the calculated variance; and
raising, by the processor, an alarm when a difference between a value of the time series predicted by the linear regression framework and an observed value of the time series for the specified time is larger than the threshold; and
allocating computing resources to process a workload based on the time series predicted by the linear regression framework, wherein the allocating includes:
identifying those anomalies in which the difference between the value of the time series predicted by the linear regression framework and the observed value of the time series is below the threshold as echoing anomalies, and
preventing, from being considered for allocating the computing resources, the echoing anomalies in the time series predicted by the linear regression framework.
2. The computer-implemented method of claim 1 , wherein the modeling includes fitting the linear regression framework using a least squares approach with a non-negative form of regularization.
3. The computer-implemented method of claim 1 , wherein the calculating includes deriving the variance as a sum of a first variance of a difference between a value of the signal portion and a value of the signal portion estimated by the linear regression framework for the time point, and a second variance.
4. The computer-implemented method of claim 3 , wherein the first variance is estimated from values of the time series determined to contain no anomalies.
5. The computer-implemented method of claim 3 , wherein
the linear regression framework is defined by a plurality of weights respectively associated with a plurality of past time periods, and
the second variance is estimated as a sum of, over a plurality of time periods, a product of, a square of the weight and a difference of, a square of, a difference between a value of the time series estimated by the linear regression framework and a value of the time series, and the first variance, for each of the time periods.
6. The computer-implemented method of claim 1 , wherein the determined threshold is five times the square root of the calculated variance.
7. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause a computer to perform a method of detecting anomalies in time series data, comprising:
instructions for modeling a time series using a linear regression framework;
instructions for representing the time series as a sum of a signal portion and a noise, wherein the signal portion corresponds to time-dependent data, and the noise removes time dependence from the data;
instructions for calculating, for a specified time, a variance of a difference between a value of the time series predicted by the linear regression framework and a value of the signal portion;
instructions for determining a threshold based on the calculated variance; and
instructions for raising an alarm when a difference between a value of the time series predicted by the linear regression framework and an observed value of the specified time is larger than the threshold; and
instructions for allocating computing resources to process a workload based on the time series predicted by the linear regression framework, wherein the allocating includes:
identifying those anomalies in which the difference between the value of the time series predicted by the linear regression framework and the observed value of the time series is below the threshold as echoing anomalies, and
preventing, from being considered for allocating the computing resources, the echoing anomalies in the time series predicted by the linear regression framework.
8. The non-transitory computer-readable storage medium of claim 7 , wherein the instructions for modeling include instructions for fitting the linear regression framework using a least squares approach with a non-negative form of regularization.
9. The non-transitory computer-readable storage medium of claim 7 , wherein the instructions for calculating include instructions for deriving the variance as a sum of a first variance of a difference between a value of the signal portion and a value of the signal portion predicted by the linear regression framework for the specified time, and a second variance.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the first variance is estimated from values of the time series determined to contain no anomalies.
11. The non-transitory computer-readable storage medium of claim 9 , wherein
the linear regression framework is defined by multiple weights respectively associated with multiple past time periods, and
the second variance is estimated as a sum of, over multiple time periods, a product of, a square of the weight and a difference of, a square of, a difference between a value of the time series estimated by the linear regression framework and a value of the time series, and the first variance, for each of the time periods.
12. The non-transitory computer-readable storage medium of claim 7 , wherein the determined threshold is five times the square root of the calculated variance.
13. A system for detecting anomalies in time series data, comprising:
a modeling component configured to model a time series using a linear regression framework;
a representing component configured to represent the time series as a sum of a signal portion and a noise, wherein the signal portion corresponds to time-dependent data, and the noise takes the data out of time dependence;
a calculating component configured to calculate a variance of a difference between a value of the time series estimated by the linear regression framework and a value of the signal portion for a time point;
a determining component configured to determine a threshold based on the calculated variance; and
an alarm component configured to raise an alarm when a difference between a value of the time series estimated by the linear regression framework and a observed value of the time series for a time point is greater than the threshold, wherein the system is further configured to allocate computing resources to process a workload based on the time series predicted by the linear regression framework, wherein the allocating includes:
identifying those anomalies in which the difference between a value of the time series predicted by the linear regression framework and the observed value of the time series is below the threshold as echoing anomalies, and
preventing, from being considered for allocating the computing resources, the echoing anomalies in the time series predicted by the linear regression framework.
14. The system of claim 13 , wherein the modeling component is further configured to model the linear regression framework using a least squares approach with a non-negative form of regularization.
15. The system of claim 13 , wherein the calculating component is further configured to derive the variance as a sum of a first variance of a difference between a value of the signal portion and a value of the signal portion estimated by the linear regression framework for the time point, and a second variance.
16. The system of claim 15 , wherein the first variance is estimated from values of the time series determined to contain no anomalies.
17. The system of claim 15 , wherein
the linear regression framework is defined by a plurality of weights respectively associated with a plurality of past time periods, and
the second variance is estimated as a sum of, over a plurality of time periods, a product of, a square of the weight and a difference of, a square of, a difference between a value of the time series estimated by the linear regression framework and a value of the time series, and the first variance, for each of the time periods.
18. The system of claim 13 , wherein the determined threshold is five times the square root of the calculated variance.