Ultra-fast pattern generation algorithm for the heterogeneous logs
A computer-implemented method for generating patterns from a set of heterogeneous log messages is presented. The method includes collecting the set of heterogenous log messages from arbitrary or unknown systems or applications or sensors or instruments, splitting the log messages into tokens based on a set of delimiters, identifying datatypes of the tokens, identifying a log structure of the log messages by generating pattern-signatures of all the tokens and the datatypes based on predefined pattern settings, generating a pattern for each of the log structures and enabling users to edit the pattern for each of the log structures based on user requirements.
1. A computer-implemented method executed on a processor for generating patterns from a set of heterogeneous log messages, the method comprising:
collecting the set of heterogenous log messages from arbitrary or unknown systems or applications or sensors or instruments;
splitting the log messages into tokens based on a set of delimiters;
identifying datatypes of the tokens;
identifying a log structure of the log messages by generating pattern-signatures of all the tokens and the datatypes based on predefined pattern settings;
generating a pattern for each of the log structures; and
enabling users to edit the pattern for each of the log structures based on user requirements.
2. The method of claim 1 , further comprising determining whether a regular expression (RegEx) rule matches any of the tokens.
3. The method of claim 2 , further comprising, if a match is determined, splitting the matched token into a plurality of tokens based on the matched RegEx rule.
4. The method of claim 3 , further comprising determining a datatype for the matched token by employing datatype RegEx syntax rules.
5. The method of claim 4 , further comprising memorizing the matched RegEx rule for the matched token based on the matched token's position within the log message.
6. The method of claim 5 , further comprising applying the memorized RegEx rule to subsequent log messages having similar position-wise tokens by employing hash indexes.
7. The method of claim 1 , further comprising maintaining an index to determine uniqueness of the pattern-signatures.
8. The method of claim 1 , wherein the predefined pattern settings are HIGH, MEDIUM, and LOW.
9. The method of claim 1 , wherein the patterns are in the form of a GROK expression.
10. A system for generating patterns from a set of heterogeneous log messages, the system comprising:
a memory; and
a processor in communication with the memory, wherein the processor runs program code to:
collect the set of heterogenous log messages from arbitrary or unknown systems or applications or sensors or instruments;
split the log messages into tokens based on a set of delimiters;
identify datatypes of the tokens;
identify a log structure of the log messages by generating pattern-signatures of all the tokens and the datatypes based on predefined pattern settings;
generate a pattern for each of the log structures; and
enable users to edit the pattern for each of the log structures based on user requirements.
11. The system of claim 10 , wherein a determination is made as to whether a regular expression (RegEx) rule matches any of the tokens.
12. The system of claim 11 , wherein, when a match is determined, the matched token is split into a plurality of tokens based on the matched RegEx rule.
13. The system of claim 12 , wherein a datatype is determined for the matched token by employing datatype RegEx syntax rules.
14. The system of claim 13 , wherein the matched RegEx rule for the matched token is memorized based on the matched token's position within the log message.
15. The system of claim 14 , wherein the memorized RegEx rule is applied to subsequent log messages having similar position-wise tokens by employing hash indexes.
16. The system of claim 10 , wherein an index is maintained to determine uniqueness of the pattern-signatures.
17. The system of claim 10 , wherein the predefined pattern settings are HIGH, MEDIUM, and LOW.
18. The system of claim 10 , wherein the patterns are in the form of a GROK expression.
19. A non-transitory computer-readable storage medium comprising a computer-readable program for generating patterns from a set of heterogeneous log messages, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
collecting the set of heterogenous log messages from arbitrary or unknown systems or applications or sensors or instruments;
splitting the log messages into tokens based on a set of delimiters;
identifying datatypes of the tokens;
identifying a log structure of the log messages by generating pattern-signatures of all the tokens and the datatypes based on predefined pattern settings;
generating a pattern for each of the log structures; and
enabling users to edit the pattern for each of the log structures based on user requirements.
20. The non-transitory computer-readable storage medium of claim 19 ,
wherein a determination is made as to whether a regular expression (RegEx) rule matches any of the tokens;
wherein, when a match is determined, the matched token is split into a plurality of tokens based on the matched RegEx rule;
wherein a datatype is determined for the matched token by employing datatype RegEx syntax rules; and
wherein the matched RegEx rule for the matched token is memorized based on the matched token's position within the log message.