Periodicity analysis on heterogeneous logs
Systems and methods are disclosed for detecting periodic event behaviors from machine generated logging by: capturing heterogeneous log messages, each log message including a time stamp and text content with one or more fields; recognizing log formats from log messages; transforming the text content into a set of time series data, one time series for each log format; during a training phase, analyzing the set of time series data and building a category model for each periodic event type in heterogeneous logs; and during live operation, applying the category model to a stream of time series data from live heterogeneous log messages and generating a flag on a time series data point violating the category model and generating an alarm report for the corresponding log message.
1. A method for detecting periodic event behaviors from machine generated logging, comprising:
capturing heterogeneous log messages, each log message including a time stamp and text content with one or more fields;
transforming the text content into a set of time series data;
during a training phase, analyzing the set of time series data and building a category model for each periodic event type in heterogeneous logs; and
during live operation, applying the category model to a stream of time series data from live heterogeneous log messages and generating a flag on a time series data point violating the category model and generating an alarm report on the corresponding log message;
the method comprising order matched messages based on the time stamps and for K ordered messages, marking their time stamps as X={X 1 , X 2 , . . . , X K } and outputting inter-arrival time series as Y={Y 1 =X 2 −X 1 , Y 2 =X 3 −X 2 , . . . , Y K-1 =X K −X K-1 }; and
estimating the categories by counting appearance times of unique values in the time series Y, sort unique values in an increasing order, and recording in an ordered list C estimated =[C 1 , C 2 , . . . C u }, and N estimated =[N 1 , N 2 , . . . N u }, where u is a number of unique values, and N i is appearance time of the unique value C i in Y.
2. The method of claim 1 , comprising clusterizing the estimated categories.
3. The method of claim 2 , comprising generating a final category model C final ={C′ 1 , C′ 2 , . . . , C′ F }, determining a model parameter (Center(C′ k ), Error(C′ k )) for each C′ k ={C i , C i+1 , . . . , C i+j }, 1≤k≤F, and a fitness score for a category model C final .
4. The method of claim 3 , comprising determining C′ m :Center(C′ m )=min{|T−Center(C′ j )|, 1≤j≤F}, where T is an inter-arrival time of log i and a previous log matching format f i with category model C f ={C′ 1 , C′ 2 , . . . , C′ F }.
5. The method of claim 4 , comprising determining a periodicity behavior violation if |T−Center(C′ m ))|>Error(C′m)).