Log parsing template generation

1. A method comprising:

receiving a plurality of logs of a computer system, wherein a log comprises information regarding an operation of the computer system;

performing a course analysis of the received plurality of logs to identify a plurality of invariants, a plurality of parameters, and a plurality of gray area terms included in the received plurality of logs, wherein the identified plurality of gray area terms is a plurality of terms included in the received plurality of logs that are undefined between one invariant associated with the identified plurality of invariants or one parameter associated with the identified plurality of parameters by the course analysis;

performing a fine analysis to identify each of the identified plurality of gray area terms as either an invariant associated with the identified plurality of invariants or a parameter associated with the identified plurality of invariants to generate two or more first log parsing templates, wherein the fine analysis is a sequence labeling analysis; and

performing a similarity analysis to combine two or more similar templates from among the two or more first log parsing templates to form two or more second log templates.

2. The method of claim 1 , wherein the similarity analysis generates a similarity value, and wherein two or more templates are deemed similar when the similarity value exceeds a threshold.

3. The method of claim 2 , wherein the similarity analysis is a Jaccard similarity analysis.

4. The method of claim 1 , wherein the course analysis includes generating a word count based on a histogram that counts each word appearing in the plurality of logs.

5. The method of claim 4 , wherein the plurality of terms identified by the histogram as appearing in the plurality of logs as a percentage of all words of the plurality of logs less than a first threshold are identified as the plurality of parameters, wherein the plurality of terms identified by the histogram as appearing in the plurality of logs as a percentage of all words of the plurality of logs greater than a second threshold are identified as the plurality of invariants, and wherein terms identified by the histogram as appearing in the plurality of logs as a percentage of all words of the plurality of logs less than a first threshold and greater than the second threshold are identified as the plurality of gray area terms.

6. The method of claim 1 , further comprising, after performing the fine analysis to identify each of the gray area terms associated with the plurality of gray area terms as either an invariant associated with the plurality of invariants or a parameter associated with the plurality of parameters, tokenizing the plurality of logs according to the identified plurality of invariants and identified plurality of parameters to generate the two or more first log parsing templates.

7. The method of claim 1 , further comprising:

parsing a new log data according to the two or more second log templates to generate a parsed log data;

analyzing the parsed log data to determine whether an anomalous event is represented in the new log data as represented by the parsed log data; and

generating and transmitting an alert when the anomalous event is determined to be represented in the new log data as represented by the parsed log data.

8. A system comprising:

a memory having instructions; and

at least one processor in communication with the memory, wherein the at least one processor is configured to:

receive a plurality of logs of a computer system, wherein a log comprises information regarding an operation of the computer system;

perform a course analysis of the received plurality of logs to identify a plurality of invariants, a plurality of parameters, and a plurality of gray area terms included in the received plurality of logs, wherein the identified plurality of gray area terms is a plurality of terms included in the received plurality of logs that are undefined between one invariant associated with the identified plurality of invariants or one parameter associated with the identified plurality of parameters by the course analysis;

perform a fine analysis to identify each of the identified plurality of gray area terms as either an invariant associated with the identified plurality of invariants or a parameter associated with the identified plurality of invariants to generate two or more first log parsing templates, wherein the fine analysis is a sequence labeling analysis; and

perform a similarity analysis to combine two or more similar templates from among the two or more first log parsing templates to form two or more second log templates, wherein the similarity analysis generates a similarity value, and wherein two or more templates are deemed similar when the similarity value exceeds a threshold.

9. The system of claim 8 , wherein the similarity analysis is a Jaccard similarity analysis.

10. The system of claim 8 , wherein the course analysis includes generating a word count based on a histogram that counts each word appearing in the plurality of logs.

11. The system of claim 10 , wherein the plurality of terms identified by the histogram as appearing in the plurality of logs as a percentage of all words of the plurality of logs less than a first threshold are identified as the plurality of parameters, wherein the plurality of terms identified by the histogram as appearing in the plurality of logs as a percentage of all words of the plurality of logs greater than a second threshold are identified as the plurality of invariants, and wherein terms identified by the histogram as appearing in the plurality of logs as a percentage of all words of the plurality of logs less than a first threshold and greater than the second threshold are identified as the plurality of gray area terms.

12. The system of claim 8 , wherein the processor is further configured to, after performing the fine analysis to identify each of the gray area terms associated with the plurality of gray area terms as either an invariant associated with the plurality of invariants or a parameter associated with the plurality of parameters, tokenize the plurality of logs according to the identified plurality of invariants and plurality of logs according to the identified plurality of invariants and identified plurality of parameters to generate the two or more first log parsing templates.

13. The system of claim 8 , wherein the processor is further configured to:

parse a new log data according to the two or more second log templates to generate parsed log data;

analyze the parsed log data to determine whether an anomalous event is represented in the new log data as represented by the parsed log data; and

generate and transmit an alert when the anomalous event is determined to be represented in the new log data as represented by the parsed log data.

14. A computer program product comprising a computer readable storage medium having program instructions executable by a processor to cause the processor to:

receive a plurality of logs of a computer system, wherein a log comprises information regarding an operation of the computer system;

perform a course analysis of the received plurality of logs to identify a plurality of invariants, a plurality of parameters, and a plurality of gray area terms included in the logs, wherein gray area terms are terms included in the received plurality of logs that are undefined between one invariant associated with the identified plurality of invariants or one parameter associated with the identified plurality of parameters by the course analysis;

perform a fine analysis to identify each of the identified plurality of gray area terms as either an invariant associated with the identified plurality of invariants or a parameter associated with the identified plurality of invariants to generate two or more first log parsing templates, wherein the fine analysis is a sequence labeling analysis; and

perform a similarity analysis to combine two or more similar templates from among the two or more first log parsing templates to form two or more second log templates.

15. The computer program product of claim 14 , wherein the similarity analysis generates a similarity value, and wherein two or more templates are deemed similar when the similarity value exceeds a threshold.

16. The computer program product of claim 15 , wherein the similarity analysis is a Jaccard similarity analysis.

17. The computer program product of claim 14 , wherein the course analysis includes generating a word count based on a histogram that counts each word appearing in the plurality of logs.

18. The computer program product of claim 17 , wherein the plurality of terms identified by the histogram as appearing in the plurality of logs as a percentage of all words of the plurality of logs less than a first threshold are identified as the plurality of parameters, wherein the plurality of terms identified by the histogram as appearing in the plurality of logs as a percentage of all words of the plurality of logs greater than a second threshold are identified as the plurality of invariants, and wherein terms identified by the histogram as appearing in the plurality of logs as a percentage of all words of the plurality of logs less than a first threshold and greater than the second threshold are identified as the plurality of gray area terms.

19. The computer program product of claim 14 , wherein the instructions executable further cause the processor to, after performing the fine analysis to identify each of the gray area terms associated with the plurality of gray area terms as either an invariant associated with the plurality of invariants or a parameter associated with the plurality of parameters, tokenize the plurality of logs according to the identified plurality of invariants and identified plurality of parameters to generate the two or more first log parsing templates.

20. The computer program product of claim 14 , wherein the instructions executable further cause the processor to:

parse a new log data according to the two or more second log templates to generate parsed log data;

analyze the parsed log data to determine whether an anomalous event is represented in the new log data as represented by the parsed log data; and

generate and transmit an alert when the anomalous event is determined to be represented in the new log data as represented by the parsed log data.