Systems and methods of anomaly detection in large scale production environments using time-series modeling
Systems and methods of anomaly detection using a generalized time-series forecasting model are disclosed. A time-series dataset is received and a forecast is generated for the time-series dataset utilizing a trained generalized time-series forecasting model configured to utilize a fixed set of characteristic features. A first anomaly determination for the time-series dataset is generated based on a comparison of the forecast for the time-series dataset to the time-series dataset. In response to the first anomaly determination identifying an anomaly, an anomaly identification notification is transmitted to an additional system associated with the anomaly identification.
1 . A system, comprising:
a non-transitory memory; and
a processor communicatively coupled to the non-transitory memory, wherein the processor is configured to read a set of instructions to:
receive a time-series dataset;
preprocess the time-series dataset by decomposing the time-series dataset using Seasonal and Trend decomposition using locally estimated scatterplot smoothing (STL) to remove seasonal variability and provide a residual component;
generate a forecast for the time-series dataset utilizing a trained generalized time-series forecasting model comprising a recurrent neural network implemented as a long short-term memory (LSTM) model configured to utilize a fixed set of characteristic features;
generate a first anomaly determination for the time-series dataset based on a comparison of the forecast to the residual component for the time-series dataset, wherein the first anomaly determination identifies a data point as anomalous when an observed value lies outside of a moving-window error-based prediction interval determined during iterative training of the LSTM model;
generate a second anomaly determination for the time-series dataset utilizing an ensemble of additional anomaly detection models which apply a first-stage majority voting process across outputs of the ensemble of additional anomaly detection models to identify an anomalous data point;
determining a final anomaly classification in a second-stage ensemble configuration by identifying a data point as anomalous if either the first anomaly determination or the majority output from the first-stage ensemble identifies the data point as anomalous; and
in response to the final anomaly determination identifying an anomaly, transmitting an anomaly identification notification to an additional system associated with the anomaly identification notification.
2 . The system of claim 1 , wherein the trained generalized time-series forecasting model is configured to utilize one or more time-based features generated from the time-series dataset.
3 . The system of claim 1 , wherein the trained generalized time-series forecasting model comprises a plurality of generalized time-series forecasting each configured to generate a forecast for time-series having a predetermined set of characteristic features.
4 . The system of claim 1 , wherein the processor is configured to, prior to transmitting the anomaly identification notification, sort the anomaly into a first bucket associated with the additional system, wherein the bucket is associated with a type of the anomaly.
5 . The system of claim 4 , wherein the anomaly identification notification is sent to a reviewer system associated with the bucket.
6 . The system of claim 5 , wherein the reviewer system is associated with at least one bucket of anomalies and is configured to receive the anomaly identification notification for anomalies assigned to the at least one bucket.
7 . The system of claim 5 , wherein the reviewer system is configured to provide feedback including an indication whether the identified anomaly was an actual anomaly or normal behavior and resolution information, and wherein the feedback is stored in a time-series database in association with time-series data that generated the anomaly identification.
8 . The system of claim 1 , wherein feedback provided by the reviewer system is incorporated into an iterative training process executed by a model training engine to update one or more trained anomaly detection models, and updated models are stored in a model store database.
9 . The system of claim 1 , wherein the reviewer system is configured to identify additional anomalous data points not identified by an anomaly detection engine and to provide the additional anomalous data points as part of the feedback stored for use in training and refinement of anomaly detection models.
10 . The system of claim 1 , wherein the anomaly identification notification is routed to the reviewer system via a frontend system or directly from an anomaly detection system.
11 . The system of claim 1 , wherein the processor is configured to determine a size of a moving window used for the error-based prediction interval during iterative training and to use the window size when identifying an anomalous data point.
12 . The system of claim 1 , wherein the processor is further configured to evaluate a trained anomaly detection model by executing an evaluation process to determine a success rate of anomaly detection prior to deployment.
13 . A computer-implemented method, comprising:
receiving a time-series dataset;
preprocessing the time-series dataset by decomposing the time-series dataset using Seasonal and Trend decomposition using locally estimated scatterplot smoothing (STL) to remove seasonal variability and provide a residual component;
generating, by a trained generalized time-series forecasting model comprising a recurrent neural network implemented as a long short-term memory (LSTM) model, a forecast for the time-series dataset using a fixed set of characteristic features;
generating a first anomaly determination for the time-series dataset based on a comparison of the forecast to the residual component for the time-series dataset, wherein the first anomaly determination identifies a data point as anomalous when an observed value lies outside of a moving-window error-based prediction interval determined during iterative training of the LSTM model;
generating a second anomaly determination for the time-series dataset utilizing an ensemble of additional anomaly detection models which apply a first-stage majority voting process across outputs of the ensemble of additional anomaly detection models to identify an anomalous data point;
determining a final anomaly classification in a second-stage ensemble configuration by identifying a data point as anomalous if either the first anomaly determination or the majority output from the first-stage ensemble identifies the data point as anomalous
in response to the final anomaly determination identifying an anomaly, transmitting an anomaly identification notification to a system associated with the anomaly identification notification.
14 . The computer-implemented method of claim 13 , wherein the generalized time-series forecasting model is configured to utilize one or more time-based features generated from the time-series dataset.
15 . The computer-implemented method of claim 13 , wherein the trained generalized time-series forecasting model comprises a plurality of generalized time-series forecasting models each configured to generate a forecast for time-series having a predetermined set of characteristic features.
16 . The computer-implemented method of claim 13 , comprising sorting the identified anomaly into a first bucket associated with the system, wherein the bucket is associated with a type of the anomaly.
17 . A computer-implemented method, comprising:
receiving a plurality of historic time-series datasets;
preprocessing each of the historic time-series dataset by decomposing the time-series dataset using Seasonal and Trend decomposition using locally estimated scatterplot smoothing (STL) to remove seasonal variability and provide a corresponding residual component;
generating a fixed set of characteristic features for each of the time-series datasets;
generating at least one time-based feature for each of the time-series datasets;
iteratively training a generalized time-series forecasting model to generate a time-series forecast based, at least in part, on the at least one characteristic feature and the at least one time-based feature generated for each of the time-series datasets, the generalized time-series forecasting model comprising a recurrent neural network implemented as a long short-term memory (LSTM) model; and
storing the generalized time-series forecasting model in a model store database.
18 . The computer-implemented method of claim 17 , wherein the fixed set of characteristic features is generated by observing a selected time-series in the plurality of historic time-series datasets over a predetermined time period.
19 . The computer-implemented method of claim 17 , wherein iteratively training the generalized time-series forecasting model comprises reducing a cost function based on a forecast for a selected one of the plurality of historic time-series datasets and the selected one of the plurality of historic time-series datasets.