Methods and systems for abnormality analysis of streamed log data
This disclosure presents systems and methods for run-time analysis of streams of log data for abnormalities using a statistical structure of meta-data associated with the log data. The systems and methods convert a log data stream into meta-data and perform statistical analysis in order to reveal a dominant statistical pattern within the meta-data. The meta-data is represented as a graph with nodes that represent each of the different event types, which are detected in the stream along with event sources associated with the events. The systems and methods use real-time analysis to compare a portion of a current log data stream collected in an operational window with historically collected meta-data represented by a graph in order to determine the degree of abnormality of the current log data stream collected in the operational window.
1. A data-processing system comprising:
one or more processors;
one or more computer-readable media; and
a routine that executes on the one or more processors to analyze a stream of digitally encoded log data output from a computational system and stored in the computer-readable media by
maintaining a historical dynamic normalcy graph of meta-data that represents statistical patterns within the stream of log data based on a first set of event types from the stream of log data;
detecting changes within the stream of log data based on similarities between the historical dynamic normalcy graph and a moving historical graph constructed from a second set of event types from the steam of log data; and
identifying abnormalities in the stream of log data based on the statistical patterns and on detected changes within the stream of log data.
2. The system of claim 1 , wherein maintaining the historical dynamic normalcy graph of meta-data further comprises:
collecting the first set of event types from the stream of log data;
calculating conditional probabilities for each pair of event types in the first set of event types;
constructing a graph of meta-data with the event types in the first set of event types as nodes and the conditional probabilities as edges; and
extracting the historical dynamic normalcy graph from the graph of meta-data.
3. The system of claim 2 , wherein extracting the dynamic normalcy graph further comprises discarding conditional probabilities that are less than a dynamic normalcy threshold.
4. The system of claim 1 , wherein detecting changes within the stream of log data further comprises:
collecting the second set of event types from the stream of log data;
calculating conditional probabilities for each pair of event types in the second set of event types;
constructing the moving historical graph of meta-data with the event types in the second set of event types as nodes and the conditional probabilities as edges;
calculating a similarity measure between the historical dynamic normalcy graph and the moving historical graph; and
replacing the historical dynamic normalcy graph with the moving historical graph when the similarity measure is less than a similarity threshold.
5. The system of claim 1 , wherein identify abnormalities in the stream of log data further comprises:
collecting event types from the stream of log data within an observation time window;
calculating a mismatch between the event types collected within the observation window and the event types represented in the historical dynamic normalcy graph;
calculating a relative abnormality degree based on the mismatch and a historical abnormality;
generating an alert when the relative abnormality degree is greater than zero; and
identifying sources of event based on associated mismatches greater than a mismatch threshold.
6. The system of claim 5 , wherein calculating a relative abnormality degree based on the mismatch and a historical abnormality further comprise:
collecting a set of mismatches for pervious time increments of the observation window; and
calculating the historical abnormality for the set of mismatches based on method of whiskers or dynamic thresholding.
7. A method carried out within a computer system having one or more processors and an electronic memory, the method comprising:
collecting first and second sets of event types from a stream of log data, the second set of event types a subset of the first set of event types;
maintaining a historical dynamic normalcy graph of meta-data that represents statistical patterns based on the first set of event types;
detecting changes within the stream of log data based on the second set of event types; and
identifying abnormalities in the stream of log data based on the statistical patterns and the detected changes and the event types collected within an observation window of the stream of log data.
8. The method of claim 7 , wherein maintaining the historical dynamic normalcy graph of meta-data further comprises:
calculating conditional probabilities for each pair of event types in the first set;
constructing a graph of meta-data with the event types in the first set of event types as nodes and the conditional probabilities as edges; and
extracting the historical dynamic normalcy graph from the graph of meta-data.
9. The method of claim 8 , wherein extracting the dynamic normalcy graph further comprises discarding conditional probabilities that are less than a dynamic normalcy threshold.
10. The method of claim 7 , wherein detecting changes within the stream of log data further comprises:
calculating conditional probabilities for each pair of event types in the second set of event types;
constructing a moving historical graph of meta-data with the event types in the second set of event types as nodes and the conditional probabilities as edges;
calculating a similarity measure between the historical dynamic normalcy graph and the moving historical graph; and
replacing the historical dynamic normalcy graph with the moving historical graph when the similarity measure is less than a similarity threshold.
11. The method of claim 7 , wherein identify abnormalities in the stream of log data further comprises:
calculating a mismatch between the event types in the first and second sets;
calculating a relative abnormality degree based on the mismatch and a historical abnormality;
generating an alert when the relative abnormality degree is greater than zero; and
identifying sources of event based on associated mismatches greater than a mismatch threshold.
12. The method of claim 11 , wherein calculating a relative abnormality degree based on the mismatch and a historical abnormality further comprise:
collecting a set of mismatches for pervious time increments of the observation window; and
calculating the historical abnormality for the set of mismatches based on method of whiskers or dynamic thresholding.
13. The method of claim 7 , wherein the first set of event types is larger than the second set of event types.
14. A non-transitory computer-readable medium encoded with machine-readable instructions that implement a method carried out by one or more processors of a computer system to perform the operations of
collecting first and second sets of event types from a stream of log data, the second set of event types a subset of the first set of event types;
maintaining a historical dynamic normalcy graph of meta-data that represents statistical patterns based on the first set of event types;
detecting changes within the stream of log data based on the second set of event types; and
identifying abnormalities in the stream of log data based on the statistical patterns and the detected changes and the event types collected within an observation window of the stream of log data.
15. The medium of claim 14 , wherein maintaining the historical graph of meta-data further comprises:
calculating conditional probabilities for each pair of event types in the first set of event types;
constructing a graph of meta-data with the event types in the first set of event types as nodes and the conditional probabilities as edges; and
extracting the historical dynamic normalcy graph from the graph of meta-data.
16. The medium of claim 15 , wherein extracting the dynamic normalcy graph further comprises discarding conditional probabilities that are less than a dynamic normalcy threshold.
17. The medium of claim 14 , wherein detecting changes within the stream of log data further comprises:
calculating conditional probabilities for each pair of event types in the second set of event types;
constructing a moving historical graph of meta-data with the event types in the second set of event types as nodes and the conditional probabilities as edges;
calculating a similarity measure between the historical dynamic normalcy graph and the moving historical graph; and
replacing the historical dynamic normalcy graph with the moving historical graph when the similarity measure is less than a similarity threshold.
18. The medium of claim 17 , wherein identify abnormalities in the stream of log data further comprises:
calculating a mismatch between the event types in the first and second sets;
calculating a relative abnormality degree based on the mismatch and a historical abnormality;
generating an alert when the relative abnormality degree is greater than zero; and
identifying sources of event based on associated mismatches greater than a mismatch threshold.
19. The medium of claim 18 , wherein calculating the relative abnormality degree based on the mismatch and a historical abnormality further comprise:
collecting a set of mismatches for pervious time increments of the observation window; and
calculating the historical abnormality for the set of mismatches based on method of whiskers or dynamic thresholding.
20. The medium of claim 19 , wherein the first set of event types is larger than the second set of event types.