Real-time anomaly detection and correlation of time-series data
The present disclosure provides systems and methods for detecting and correlating anomalous time-series data. A system may receive and process time-series data associated with one or more network data streams to generate sets of aligned time-series data. The system may detect anomalous time-stamped data points in the sets of aligned time series data and generate groups of annotated time-series data. The annotation identifies specific time-stamped data points as anomalous. The system may determine the number of anomalous groups of annotated time-series data within all groups of annotated time-series data and may further determine the probability that one or more anomalous groups belong to at least one of the groups of annotated time-series data using a generative statistical model and outputting one or more correlated anomalous groups. The system may generate a detailed statistical report for each correlated anomalous group and output an aggregated statistical report for the correlated groups.
1. A method for detecting and correlating anomalous time-series data, the method comprising:
detecting anomalous time-stamped data points in each of one or more sets of aligned time-series data of a network of network entities;
annotating each of the anomalous time-stamped data points;
generating, based on the annotated time-stamped data points, one or more groups of annotated time-series data;
determining, based on the annotated time-series data, a number of anomalous groups of annotated time-series data within the one or more groups of annotated time-series data;
determining, based at least on the number of anomalous groups and for each anomalous group of annotated time-series data, a probability that the anomalous group belongs to at least one of the one or more groups of annotated time-series data;
determining, based on the determined probabilities, one or more correlated anomalous groups within the anomalous groups; and
identifying at least one network entity corresponding to data in the one or more correlated anomalous groups.
2. The method of claim 1 , wherein detecting the anomalous time-stamped data points in each of the one or more sets of aligned time-series data is performed using one or more virtual interfaces, wherein the one or more virtual interfaces are determined based on a data type of each of the one or more sets of the aligned time-series data.
3. The method of claim 2 , wherein the one or more of virtual interfaces may detect the anomalous time-stamped data points in parallel.
4. The method of claim 1 , wherein detecting the anomalous time-stamped data points includes discretizing each of the one or more sets of aligned time-series.
5. The method of claim 1 , wherein determining the anomalous groups within the one or more groups of annotated sets of time-series data is performed using a locality sensitive hashing algorithm.
6. The method of claim 1 , wherein determining the probability that the anomalous groups belongs to at least one of the one or more groups of annotated time-series data and determining the one or more correlated anomalous groups is performed in parallel.
7. The method of claim 1 , further comprising:
generating a detailed statistical report for each of the one or more correlated anomalous groups;
aggregating the detailed statistical reports for each of the one or more correlated anomalous groups; and
outputting the aggregated statistical report.
8. A system for detecting and correlating anomalous time-series data, the system comprising:
a memory storing one or more generative statistical models; and
one or more processors coupled to the memory, the one or more processors configured to:
detect anomalous time-stamped data points in each of one or more sets of aligned time-series data of a network of network entities;
annotate each of the anomalous time-stamped data points;
generate, based on the annotated time-stamped data points, one or more groups of annotated time-series data;
determine, based on the annotated time-series data, a number of anomalous groups of annotated time-series data within the one or more groups of annotated time-series data;
determine, based at least on the number of anomalous groups and for each anomalous group of annotated time-series data, a probability that the anomalous group belongs to at least one of the one or more groups of annotated time-series data;
determine, based on the determined probabilities, one or more correlated anomalous groups within the anomalous groups; and
identify at least one network entity corresponding to data in the one or more correlated anomalous groups.
9. The system of claim 8 , further comprising one or more virtual interfaces, wherein detecting anomalous time-stamped data points in each of the one or more sets of aligned time-series data is performed using the one or more virtual interfaces, wherein the one or more virtual interfaces are determined based on a data type of each of the one or more sets of the aligned time-series data.
10. The system of claim 9 , wherein the one or more of virtual interfaces may detect the anomalous time-stamped data points in parallel.
11. The system of claim 8 , wherein detecting anomalous time-stamped data points includes discretizing, by the one or more processors, each of the one or more sets of aligned time-series data.
12. The system of claim 8 , wherein determining the anomalous groups within the one or more groups of annotated sets of time-series data is performed using a locality sensitive hashing algorithm.
13. The system of claim 8 , wherein determining the probability that the anomalous groups belongs to at least one of the one or more groups of annotated time-series data and determining the one or more correlated anomalous groups is performed in parallel.
14. The system of claim 8 , wherein the one or more processors are further configured to:
generate a detailed statistical report for each of the one or more correlated anomalous group;
aggregate the detailed statistical reports for each of the one or more correlated anomalous group; and
output the aggregated statistical report.
15. A non-transitory computer readable storage medium storing instructions executable by a processor for performing a method comprising:
detecting anomalous time-stamped data points in each of one or more sets of aligned time-series data of a network of network entities;
annotating each of the anomalous time-stamped data points;
generating, based on the annotated time-stamped data points, one or more groups of annotated time-series data;
determining, based on the annotated time-series data, a number of anomalous groups of annotated time-series data within the one or more groups of annotated time-series data;
determining, based at least on the number of anomalous groups and for each anomalous group of annotated time-series data, a probability that the anomalous group belongs to at least one of the one or more groups of annotated time-series data;
determining, based on the determined probabilities, one or more correlated anomalous groups within the anomalous groups;
identifying at least one network entity corresponding to data in the one or more correlated anomalous groups.
16. The non-transitory computer readable storage medium of claim 15 , wherein detecting the anomalous time-stamped data points in each of the one or more sets of aligned time-series data is performed using one or more virtual interfaces, wherein the one or more virtual interfaces are determined based on a data type of each of the one or more sets of the aligned time-series data.
17. The non-transitory computer readable storage medium of claim 16 , wherein the one or more of virtual interfaces may detect the anomalous time-stamped data points in parallel.
18. The non-transitory computer readable storage medium of claim 15 , wherein detecting anomalous time-stamped data points includes discretizing each of the one or more sets of aligned time-series data.
19. The non-transitory computer readable storage medium of claim 15 , wherein determining the anomalous groups within the one or more groups of annotated sets of time-series data is performed using a locality sensitive hashing algorithm.
20. The non-transitory computer readable storage medium of claim 15 , wherein determining the probability that the anomalous groups belongs to at least one of the one or more groups of annotated time-series data and determining the one or more correlated anomalous groups is performed in parallel.