Anomaly detection data workflow for time series data
Methods and systems are described herein for improving anomaly detection in timeseries datasets. Different machine learning models may be trained to process specific types of timeseries data efficiently and accurately. Thus, selecting a proper machine learning model for identifying anomalies in a specific set of timeseries data may greatly improve accuracy and efficiency of anomaly detection. Another way to improve anomaly detection is to process a multitude of timeseries datasets for a time period (e.g., 90 days) to detect anomalies from those timeseries datasets and then correlate those detected anomalies by generating an anomaly timeseries dataset and identifying anomalies within the anomaly timeseries dataset. Yet another way to improve anomaly detection is to divide a dataset into multiple datasets based on a type of anomaly detection requested.
1 . A system for correlating events based on multiple anomalies occurring within a given time interval across multiple timeseries datasets, the system comprising:
one or more processors; and
a non-transitory computer-readable storage medium storing instructions, which when executed by the one or more processors cause the one or more processors to:
determine a temporal trait associated with a timeseries dataset of a plurality of timeseries datasets, the temporal trait identifying a pattern within the timeseries dataset;
select, from a first plurality of anomaly detection models and based on determining the temporal trait, a first anomaly detection model, wherein the first plurality of anomaly detection models are trained to detect anomalies in datasets classified under temporal traits;
determine a model execution parameter of a plurality of model execution parameters for the first anomaly detection model based on a timeseries signal associated with the timeseries dataset;
obtain, based on selecting the first anomaly detection model and determining the model execution parameter, a plurality of sets of timestamps by inputting the timeseries dataset into the first anomaly detection model, wherein each set of the plurality of sets of the timestamps includes a corresponding plurality of timestamps, each timestamp representing an anomaly detected within the timeseries dataset;
generate, based on the plurality of sets of the timestamps, an anomaly timeseries dataset having a corresponding aggregated value representing the timestamps for a plurality of time intervals;
input the anomaly timeseries dataset into a second anomaly detection model trained to detect anomalies in aggregated datasets to obtain one or more anomalies, wherein the anomaly timeseries dataset stores aggregated values generated from the plurality of sets of the timestamps that represent the anomalies within the plurality of sets of the timestamps, and wherein the second anomaly detection model is not one of the first plurality of anomaly detection models; and
generate a message based on the one or more anomalies.
2 . The system of claim 1 , wherein the instructions further cause the one or more processors to:
receive the plurality of timeseries datasets,
wherein each timeseries dataset of the plurality of timeseries datasets comprises a plurality of values for a plurality of timestamps,
wherein the plurality of timeseries datasets includes a first dataset comprising a first type of data and a second dataset comprising a second type of data,
wherein the timeseries dataset is the first dataset,
wherein, the plurality of sets of timestamps are obtained by inputting each of the plurality of timeseries datasets into at least one anomaly detection model of the first plurality of anomaly detection models, and
wherein each set of the plurality of sets of timestamps comprises one or more timestamps representing the one or more anomalies detected within the timeseries dataset.
3 . The system of claim 2 , wherein the instructions that cause the one or more processors to select the first anomaly detection model cause the one or more processors to:
select, based on the first type of data, the first anomaly detection model for the first dataset; and
select, based on the second type of data, a third anomaly detection model for the second dataset,
wherein the first anomaly detection model and the third anomaly detection model are of the first plurality of anomaly detection models,
wherein the first dataset is inputted into the first anomaly detection model, and
wherein the second dataset is inputted into the third anomaly detection model.
4 . The system of claim 1 , wherein the instructions further cause the one or more processors to:
combine the timestamps within the plurality of sets of the timestamps into a chronologically ordered dataset of the anomalies by:
retrieving a time interval;
retrieving, from the chronologically ordered dataset, a time associated with a first timestamp stored in a first position within the chronologically ordered dataset;
traversing the chronologically ordered dataset until a second timestamp is reached, wherein the second timestamp is a last timestamp within a time slot associated with the first timestamp; and
generating a first aggregated value based on all the timestamps starting from the first timestamp and ending with the second timestamp, wherein the first aggregated value represents a count of the anomalies detected starting with the first timestamp and ending with the second timestamp.
5 . The system of claim 1 , wherein the instructions further cause the one or more processors to:
receive, from the first anomaly detection model and based on the anomaly timeseries dataset, one or more probabilities corresponding to the one or more anomalies detected by the first anomaly detection model;
retrieve an anomaly confidence threshold; and
remove from the one or more anomalies those anomalies that do not meet the anomaly confidence threshold.
6 . A method comprising:
determining a temporal trait associated with a timeseries dataset of a plurality of timeseries datasets, the temporal trait identifying a pattern within the timeseries dataset;
selecting, from a first plurality of anomaly detection models and based on determining the temporal trait, a first anomaly detection model trained to detect anomalies in datasets classified under the temporal trait;
determining a model execution parameter of a plurality of model execution parameters for the first anomaly detection model based on a timeseries signal associated with the timeseries dataset;
receiving, based on selecting the first anomaly detection model and determining the model execution parameter, a plurality of sets of timestamps by inputting the timeseries dataset into the first anomaly detection model, wherein each set of the plurality of sets of timestamps includes a corresponding plurality of timestamps, each timestamp representing an anomaly detected within the timeseries dataset;
generating, based on the plurality of sets of the timestamps, an anomaly timeseries dataset having a corresponding aggregated value representing timestamps for a plurality of time intervals;
inputting the anomaly timeseries dataset into a second anomaly detection model to obtain one or more anomalies, wherein the anomaly timeseries dataset stores aggregated values generated from the plurality of sets of timestamps that represent anomalies within the plurality of sets of timestamps, and wherein the second anomaly detection model is not one of the first plurality of anomaly detection models; and
generating a message based on the one or more anomalies.
7 . The method of claim 6 , further comprising:
receiving the plurality of timeseries datasets, wherein each timeseries dataset comprises a plurality of values.
8 . The method of claim 7 , wherein the plurality of timeseries datasets includes a first dataset comprising a first type of data and a second dataset comprising a second type of data,
wherein the timeseries dataset is the first dataset.
9 . The method of claim 8 , comprising:
selecting, based on the first type of data, the first anomaly detection model of the first plurality of anomaly detection models; and
selecting, based on the second type of data, a third anomaly detection model of the first plurality of anomaly detection models.
10 . The method of claim 6 , further comprising combining the timestamps within the plurality of sets of timestamps into a chronologically ordered dataset of anomalies by:
storing, in a data structure, a first set of the plurality of sets of timestamps in a chronological order;
selecting each set of the plurality of sets of timestamps; and
placing each timestamp from each selected set into the data structure in the chronological order.
11 . The method of claim 10 , wherein transforming the plurality of sets of timestamps into the anomaly timeseries dataset comprises:
retrieving a time interval;
retrieving, from the data structure, a time associated with a first timestamp stored in a first position within the data structure;
traversing the data structure until a second timestamp is reached, wherein the second timestamp is a last timestamp within a time slot associated with the first timestamp; and
generating a first aggregated value based on all the timestamps between the first timestamp and the second timestamp, wherein the first aggregated value represents a count of the anomalies detected between the first timestamp and the second timestamp.
12 . The method of claim 6 , further comprising:
receiving, from the first anomaly detection model and based on the anomaly timeseries dataset, one or more probabilities corresponding to the one or more anomalies detected by the first anomaly detection model;
retrieving an anomaly confidence threshold; and
removing from the one or more anomalies those anomalies that do not meet the anomaly confidence threshold.
13 . One or more non-transitory, computer-readable media for correlating events based on anomalies occurring within a given time interval across multiple timeseries datasets, storing instructions that, when executed by one or more processors, cause operations comprising:
determining a temporal trait associated with a timeseries dataset of a plurality of timeseries datasets, the temporal trait identifying a pattern within the timeseries dataset;
selecting, from a first plurality of anomaly detection models and based on determining the temporal trait, a first anomaly detection model trained to detect anomalies in datasets classified under the temporal trait;
determining a model execution parameter of a plurality of model execution parameters for the first anomaly detection model based on a timeseries signal associated with the timeseries dataset;
receiving, based on selecting the first anomaly detection model and determining the model execution parameter, a plurality of sets of timestamps by inputting the timeseries dataset into the first anomaly detection model, wherein each set of the plurality of sets of timestamps includes a plurality of timestamps, each timestamp representing an anomaly detected within the timeseries dataset;
generating, based on the plurality of sets of the timestamps, an anomaly timeseries dataset having a corresponding aggregated value representing timestamps for a plurality of time intervals;
inputting the anomaly timeseries dataset into a second anomaly detection model to obtain one or more anomalies, wherein the anomaly timeseries dataset stores aggregated values generated from the plurality of sets of timestamps that represent anomalies within the plurality of sets of timestamps, and wherein the second anomaly detection model is not one of the first plurality of anomaly detection models; and
generating a message based on the one or more anomalies.
14 . The one or more non-transitory, computer-readable media of claim 13 , wherein the instructions further cause the one or more processors to:
receive the plurality of timeseries datasets, wherein each timeseries dataset comprises a plurality of values for the plurality of timestamps.
15 . The one or more non-transitory, computer-readable media of claim 14 , wherein the plurality of timeseries datasets includes a first dataset comprising a first type of data and a second dataset comprising a second type of data,
wherein the timeseries dataset is the first dataset.
16 . The one or more non-transitory, computer-readable media of claim 15 , wherein the instructions further cause the one or more processors to:
inputting the first dataset into the first anomaly detection model of the first plurality of anomaly detection models; and
inputting the second dataset into a third anomaly detection model of the first plurality of anomaly detection models.
17 . The one or more non-transitory, computer-readable media of claim 13 , further comprising instructions that cause the one or more processors to combine the timestamps within the plurality of sets of timestamps into a chronologically ordered dataset of the anomalies by:
storing, in a data structure, a first set of the plurality of sets of timestamps in a chronological order;
selecting each set of the plurality of sets of timestamps; and
placing each timestamp from each selected set into the data structure in the chronological order.
18 . The one or more non-transitory, computer-readable media of claim 17 , wherein the instructions that cause the one or more processors to transform the plurality of sets of timestamps into the anomaly timeseries dataset further cause the one or more processors to:
retrieve a time interval;
retrieve, from the data structure, a time associated with a first timestamp stored in a first position within the data structure;
traverse the data structure until a second timestamp is reached, wherein the second timestamp is a last timestamp within a time slot associated with the first timestamp; and
generate a first aggregated value based on all the timestamps between the first timestamp and the second timestamp, wherein the first aggregated value represents a count of the anomalies detected between the first timestamp and the second timestamp.
19 . The one or more non-transitory, computer-readable media of claim 13 , wherein the instructions further cause the one or more processors to:
receive, from the first anomaly detection model and based on the anomaly timeseries dataset, one or more probabilities corresponding to the one or more anomalies detected by the first anomaly detection model;
retrieve an anomaly confidence threshold; and
remove from the one or more anomalies those anomalies that do not meet the anomaly confidence threshold.
20 . The one or more non-transitory, computer-readable media of claim 13 , wherein the instructions that cause the one or more processors to transform the plurality of sets of timestamps into the anomaly timeseries dataset further cause the one or more processors to generate the anomaly timeseries dataset with a plurality of time windows and a corresponding number of anomalies detected during a corresponding time window.