Machine learning time series anomaly detection
A method includes receiving a time series anomaly detection query from a user and training one or more models using a set of time series data values. For each respective time series data value in the set, the method includes determining, using the trained models, an expected data value for the respective time series data value and determining a difference between the expected data value and the respective time series data value. The method also includes determining that the difference between the expected data value and the respective time series data value satisfies a threshold. In response to determining that the difference between the expected data value and the respective time series data value satisfies the threshold, the method includes determining that the respective time series data value is anomalous and reporting the anomalous respective time series data value to the user.
1 . A computer-implemented method comprising:
receiving, by data processing hardware, a time series anomaly detection query from a user, the time series anomaly detection query requesting the data processing hardware to determine one or more anomalies in a set of time series data values;
training, by the data processing hardware, a chain of two or more models using each time series data value in the set of time series data values, wherein the two or more models includes a separate model for each component from a plurality of components to improve scalability by at least:
training, by the data processing hardware, a first model of the chain to determine a holiday effect component for the set of time series data values;
removing, by the data processing hardware, the holiday effect component from the time series data values to generate first intermediate data;
training, by the data processing hardware, an outlier model of the chain using the first intermediate data to determine an outlier component
removing, by the data processing hardware, the outlier component from the first intermediate data to generate second intermediate data; and
training, by the data processing hardware, a second model of the chain using the second intermediate data to determine a seasonal component and a trend component; and
for each respective time series data value in the set of time series data values:
determining, by the data processing hardware and using the chain of two or more models, an expected data value for the respective time series data value by at least:
decomposing, using the chain of two or more models, the respective time series data value into the plurality of components, wherein each model from the chain of two or more models generates one component of the plurality of components; and
aggregating two or more components of the plurality of components into the expected data value by summing at least the trend component, the holiday effect component, and the seasonal component;
determining, by the data processing hardware, a difference between the expected data value for the respective time series data value and the respective time series data value;
determining, by the data processing hardware, whether the difference between the expected data value for the respective time series data value and the respective time series data value satisfies a threshold;
in response to determining that the difference between the expected data value for the respective time series data value and the respective time series data value satisfies the threshold, determining, by the data processing hardware, that the respective time series data value is anomalous; and
reporting, by the data processing hardware, the respective time series data value to the user.
2 . The method of claim 1 , wherein the plurality of components comprises one or more of a trend component, a holiday effect component, a seasonal component, and a step change component.
3 . The method of claim 1 , wherein the threshold is based on an anomaly probability received from the user.
4 . The method of claim 1 , wherein the time series anomaly detection query comprises a single Structured Query Language (SQL) query.
5 . The method of claim 4 , wherein the single SQL query requests the data processing hardware to determine one or more anomalous data values in a plurality of sets of time series data values.
6 . The method of claim 1 , wherein the chain of two or more models comprises a plurality of auto regressive integrated moving average models with different orders of autoregressive models, different degrees of differencing, and an order of a moving-average model.
7 . The method of claim 1 , further comprising:
forecasting, using the chain of two or more models, a future time series data value;
after forecasting the future time series data value, receiving an additional time series data value associated with the future time series data value;
determining a difference between the future time series data value and the additional time series data value;
determining whether the difference between the future time series data value and the additional time series data value satisfies the threshold;
in response to determining that the difference between the future time series data value and the additional time series data value satisfies the threshold, determining that the additional time series data value is anomalous; and
reporting the additional time series data value to the user.
8 . The method of claim 1 , wherein the set of time series data values comprises a set of univariate time series data values.
9 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to:
receive a time series anomaly detection query from a user, the time series anomaly detection query requesting the data processing hardware to determine one or more anomalies in a set of time series data values;
train a chain of two or more models using each time series data value in the set of time series data values, wherein the two or more models includes a separate model for each component from a plurality of components to improve scalability by at least:
training a first model of the chain to determine a holiday effect component for the set of time series data values;
removing the holiday effect component from the time series data values to generate first intermediate data;
training an outlier model of the chain using the first intermediate data to determine an outlier component
removing the outlier component from the first intermediate data to generate second intermediate data; and
training a second model of the chain using the intermediate second intermediate data to determine a seasonal component and a trend component; and
for each respective time series data value in the set of time series data values:
determine, using the chain of two or more models, an expected data value for the respective time series data value by at least:
decomposing, using the chain of two or more models, the respective time series data value into the plurality of components, wherein each model from the chain of two or more models generates one component of the plurality of components; and
aggregating two or more components of the plurality of components into the expected data value by summing at least the trend component, the holiday effect component, and the seasonal component;
determine a difference between the expected data value for the respective time series data value and the respective time series data value;
determine whether the difference between the expected data value for the respective time series data value and the respective time series data value satisfies a threshold;
in response to determining that the difference between the expected data value for the respective time series data value and the respective time series data value satisfies the threshold, determine that the respective time series data value is anomalous; and
report the respective time series data value to the user.
10 . The system of claim 9 , wherein the plurality of components comprises one or more of a trend component, a holiday effect component, a seasonal component, and a step change component.
11 . The system of claim 9 , wherein the threshold is based on an anomaly probability received from the user.
12 . The system of claim 9 , wherein the time series anomaly detection query comprises a single Structured Query Language (SQL) query.
13 . The system of claim 12 , wherein the single SQL query requests the data processing hardware to determine one or more anomalous data values in a plurality of sets of time series data values.
14 . The system of claim 9 , wherein the chain of two or more models comprises a plurality of auto regressive integrated moving average models with different orders of autoregressive models, different degrees of differencing, and an order of a moving-average model.
15 . The system of claim 9 , wherein the instructions further cause the data processing hardware to:
forecast, using the chain of two or more models, a future time series data value;
after forecasting the future time series data value, receive an additional time series data value associated with the future time series data value;
determine a difference between the future time series data value and the additional time series data value;
determine whether the difference between the future time series data value and the additional time series data value satisfies the threshold;
in response to determining that the difference between the future time series data value and the additional time series data value satisfies the threshold, determine that the additional time series data value is anomalous; and
report the additional time series data value to the user.
16 . The system of claim 9 , wherein the set of time series data values comprises a set of univariate time series data values.