Data-driven reduction of log message data
Techniques are provided for data-driven reduction of log message data. An exemplary method comprises: obtaining log files and user-specified configuration parameters, wherein the log files each comprise one or more log messages; generating an event count matrix indicating a number of times each of a plurality of unique messages appeared in a given log file of the log files; generating a correlation graph by inserting similar messages with a mutual undirected edge, wherein similar messages are identified based on a predefined similarity measure; extracting redundant messages from the correlation graph by selecting log messages for inclusion in an uninformative log message filter from sub-graphs of the correlation graph in which any two nodes are connected together, except those log messages satisfying a predefined message frequency criteria; and identifying one or more redundant messages using the uninformative log message filter. The uninformative log message filter is optionally applied to real-time log messages and/or existing file systems.
1. A method, comprising:
obtaining a plurality of log files and one or more user-specified configuration parameters, wherein said plurality of log files each comprise one or more log messages;
generating, using at least one processing device, an event count matrix indicating a number of times each of a plurality of unique messages appeared in a given log file of the plurality of log files, wherein the step of generating the event count matrix further comprises the steps of parsing the plurality of log files, using one or more regular expressions from the one or more user-specified configuration parameters to generate an identifier for each of the plurality of log messages, into a set of dictionaries indicating a number of times each unique message has appeared in each log file; and converting the set of dictionaries into the event count matrix indicating the number of times each unique message appeared in the given log file of the plurality of log files;
generating, using the at least one processing device, a correlation graph by inserting similar messages with a mutual undirected edge using the event count matrix, wherein said similar messages are identified based on a predefined similarity measure;
extracting, using said at least one processing device, redundant messages from the correlation graph by selecting log messages for inclusion in an uninformative log message filter from sub-graphs of the correlation graph in which any two nodes are connected together, except those log messages satisfying one or more predefined message frequency criteria; and
identifying, using said at least one processing device, one or more redundant messages using said uninformative log message filter.
2. The method of claim 1 , wherein the uninformative log message filter comprises a plurality of sub-filters distributed over one or more computing nodes.
3. The method of claim 2 , further comprising the step of aggregating an output of each of said plurality of sub-filters.
4. The method of claim 1 , wherein the uninformative log message filter comprises a list of unique log messages to be discarded.
5. The method of claim 1 , wherein said one or more user-specified configuration parameters comprise one or more of a number of log files to be sampled, a lower bound of correlation between log messages when deciding whether to include the log messages in the correlation graph, and specific formats that should not be disposed even when the specific formats are redundant.
6. The method of claim 1 , wherein the uninformative log message filter is applied to one or more of real-time log messages and one or more existing file systems comprising a plurality of log files.
7. The method of claim 1 , wherein the one or more predefined message frequency criteria comprises one or more message frequency criteria based on a number of log files that include a given message.
8. A system, comprising:
a memory; and
at least one processing device, coupled to the memory, operative to implement the following steps:
obtaining a plurality of log files and one or more user-specified configuration parameters, wherein said plurality of log files each comprise one or more log messages;
generating, using at least one processing device, an event count matrix indicating a number of times each of a plurality of unique messages appeared in a given log file of the plurality of log files, wherein the step of generating the event count matrix further comprises the steps of parsing the plurality of log files, using one or more regular expressions from the one or more user-specified configuration parameters to generate an identifier for each of the plurality of log messages, into a set of dictionaries indicating a number of times each unique message has appeared in each log file; and converting the set of dictionaries into the event count matrix indicating the number of times each unique message appeared in the given log file of the plurality of log files;
generating, using the at least one processing device, a correlation graph by inserting similar messages with a mutual undirected edge using the event count matrix, wherein said similar messages are identified based on a predefined similarity measure;
extracting, using said at least one processing device, redundant messages from the correlation graph by selecting log messages for inclusion in an uninformative log message filter from sub-graphs of the correlation graph in which any two nodes are connected together, except those log messages satisfying one or more predefined message frequency criteria; and
identifying, using said at least one processing device, one or more redundant messages using said uninformative log message filter.
9. The system of claim 8 , wherein the uninformative log message filter comprises a plurality of sub-filters distributed over one or more computing nodes.
10. The system of claim 8 , wherein said one or more user-specified configuration parameters comprise one or more of a number of log files to be sampled, a lower bound of correlation between log messages when deciding whether to include the log messages in the correlation graph, and specific formats that should not be disposed even when the specific formats are redundant.
11. The system of claim 8 , wherein the uninformative log message filter is applied to one or more of real-time log messages and one or more existing file systems comprising a plurality of log files.
12. The system of claim 8 , wherein the one or more predefined message frequency criteria comprises one or more message frequency criteria based on a number of log files that include a given message.
13. A computer program product, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:
obtaining a plurality of log files and one or more user-specified configuration parameters, wherein said plurality of log files each comprise one or more log messages;
generating, using at least one processing device, an event count matrix indicating a number of times each of a plurality of unique messages appeared in a given log file of the plurality of log files, wherein the step of generating the event count matrix further comprises the steps of parsing the plurality of log files, using one or more regular expressions from the one or more user-specified configuration parameters to generate an identifier for each of the plurality of log messages, into a set of dictionaries indicating a number of times each unique message has appeared in each log file; and converting the set of dictionaries into the event count matrix indicating the number of times each unique message appeared in the given log file of the plurality of log files;
generating, using the at least one processing device, a correlation graph by inserting similar messages with a mutual undirected edge using the event count matrix, wherein said similar messages are identified based on a predefined similarity measure;
extracting, using said at least one processing device, redundant messages from the correlation graph by selecting log messages for inclusion in an uninformative log message filter from sub-graphs of the correlation graph in which any two nodes are connected together, except those log messages satisfying one or more predefined message frequency criteria; and
identifying, using said at least one processing device, one or more redundant messages using said uninformative log message filter.
14. The computer program product of claim 13 , wherein the uninformative log message filter comprises a plurality of sub-filters distributed over one or more computing nodes, wherein an output of each of said plurality of sub-filters is aggregated.
15. The computer program product of claim 13 , wherein said one or more user-specified configuration parameters comprise one or more of a number of log files to be sampled, a lower bound of correlation between log messages when deciding whether to include the log messages in the correlation graph, and specific formats that should not be disposed even when the specific formats are redundant.
16. The computer program product of claim 13 , wherein the uninformative log message filter is applied to one or more of real-time log messages and one or more existing file systems comprising a plurality of log files.
17. The computer program product of claim 13 , wherein the one or more predefined message frequency criteria comprises one or more message frequency criteria based on a number of log files that include a given message.
18. The system of claim 9 , further comprising the step of aggregating an output of each of said plurality of sub-filters.
19. The system of claim 8 , wherein the uninformative log message filter comprises a list of unique log messages to be discarded.
20. The computer program product of claim 13 , wherein the uninformative log message filter comprises a list of unique log messages to be discarded.