Structure-level anomaly detection for unstructured logs
A computer-implemented method, computer program product, and computer processing system are provided. The method includes preprocessing, by a processor, a set of heterogeneous logs by splitting each of the logs into tokens to obtain preprocessed logs. Each of the logs in the set is associated with a timestamp and textual content in one or more fields. The method further includes generating, by the processor, a set of regular expressions from the preprocessed logs. The method also includes performing, by the processor, an unsupervised parsing operation by applying the regular expressions to the preprocessed logs to obtain a set of parsed logs and a set of unparsed logs, if any. The method additionally includes storing, by the processor, the set of parsed logs in a log analytics database and the set of unparsed logs in a debugging database.
1. A computer-implemented method, comprising:
preprocessing, by a processor, a set of heterogeneous logs by splitting each of the logs into tokens to obtain preprocessed logs, each of the logs in the set associated with a timestamp and textual content in one or more fields;
generating, by the processor, a set of regular expressions from the preprocessed logs;
performing, by the processor, an unsupervised parsing operation by applying the regular expressions to the preprocessed logs to obtain a set of parsed logs and a set of unparsed logs, if any; and
storing, by the processor, the set of parsed logs in a log analytics database and the set of unparsed logs in a debugging database;
wherein the unsupervised parsing operation comprises identifying timestamp fields in the set of heterogeneous logs using a set of predefined datetime formats that are based on the regular expressions; and
wherein any of the timestamp fields matching any of the predefined datetime formats are cached and initially used for subsequent matching prior to uncached ones of the predefined datetime formats.
2. The computer-implemented method of claim 1 , wherein the cached ones of the predefined datetime formats are stored in a cache in a descending length order.
3. A computer-implemented method, comprising:
preprocessing, by a processor, a set of heterogeneous logs by splitting each of the logs into tokens to obtain preprocessed logs, each of the logs in the set associated with a timestamp and textual content in one or more fields;
generating, by the processor, a set of regular expressions from the preprocessed logs;
performing, by the processor, an unsupervised parsing operation by applying the regular expressions to the preprocessed logs to obtain a set of parsed logs and a set of unparsed logs, if any; and
storing, by the processor, the set of parsed logs in a log analytics database and the set of unparsed logs in a debugging database;
wherein said parsing operation parses a particular one of the preprocessed logs by:
generating a log signature from the particular one of the preprocessed logs;
matching the regular expressions to the log signature to determine whether a match exists there between;
parsing the log signature using any of the regular expressions matching thereto, responsive to the match existing; and
generating a new regular expression to parse the particular one of the preprocessed logs, responsive to a lack of the match existing.
4. The computer-implemented method of claim 3 , wherein the log signature is generated from the particular one of the preprocessed logs by concatenating data types of the tokens of the particular one of the preprocessed logs.
5. A computer program product, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
preprocessing, by a processor, a set of heterogeneous logs by splitting each of the logs into tokens to obtain preprocessed logs, each of the logs in the set associated with a timestamp and textual content in one or more fields;
generating, by the processor, a set of regular expressions from the preprocessed logs;
performing, by the processor, an unsupervised parsing operation by applying the regular expressions to the preprocessed logs to obtain a set of parsed logs and a set of unparsed logs, if any; and
storing, by the processor, the set of parsed logs in a log analytics database and the set of unparsed logs in a debugging database;
wherein the unsupervised parsing operation comprises identifying timestamp fields in the set of heterogeneous logs using a set of predefined datetime formats that are based on the regular expressions; and
wherein any of the timestamp fields matching any of the predefined datetime formats are cached and initially used for subsequent matching prior to uncached ones of the predefined datetime formats.
6. The computer program product of claim 5 , wherein the cached ones of the predefined datetime formats are stored in a cache in a descending length order.