Compression of log data using pattern recognition
Techniques are provided for compression and decompression of log data using pattern recognition. An exemplary method comprises: obtaining at least one log message, wherein the at least one log message comprises a pattern-based message template and one or more message variables, wherein the pattern-based message template was learned from historical log data using a log pattern mining technique; obtaining a compression index that maps a plurality of pattern-based message templates to a corresponding message signature; and writing the one or more message variables and a message signature corresponding to the pattern-based message template of the at least one log message to a log file. The log pattern mining technique learns a given pattern-based message template from the historical log data to represent a plurality of historical log messages. The at least one log message may comprise a historical log message or may be from a real-time stream of log messages.
1. A method, comprising:
learning at least one pattern that is repeated across a plurality of log messages in historical log data using a log pattern mining technique that learns the at least one pattern to represent a plurality of historical log messages in the historical log data;
generating a plurality of different pattern-based message templates based at least in part on the learned at least one pattern;
generating a compression index that maps each of the plurality of different pattern-based message templates to a corresponding message signature of a plurality of message signatures; and
performing the following steps, subsequent to the generating the compression index, in response to obtaining at least one log message, wherein the at least one log message comprises at least one pattern-based message template and one or more message variable values, wherein the at least one pattern-based message template comprises at least one portion that is replaced with at least one of the message variable values:
separating the at least one log message into the at least one pattern-based message template and the one or more message variable values using the learned at least one pattern;
obtaining the compression index, generated prior to the obtaining, that maps each of the plurality of different pattern-based message templates to the corresponding message signature of the plurality of message signatures; and
writing, using at least one processing device, the one or more separated message variable values and a message signature, obtained from the compression index, corresponding to the separated at least one pattern-based message template of the at least one log message to a log file.
2. The method of claim 1 , further comprising initializing a counter associated with the at least one pattern-based message template in response to the at least one pattern-based message template not being in the compression index and incrementing the counter associated with the at least one pattern-based message template in response to the at least one pattern-based message template being in the compression index, wherein a given message signature is assigned to a particular pattern-based message template based on a length of the given message signature and a frequency of occurrence of the particular pattern-based message template.
3. The method of claim 2 , further comprising, when a given pattern-based message template is not in the compression index: adding a message signature to the compression index and a decompression index for the given pattern-based message template.
4. The method of claim 1 , wherein the compression index comprises a key-value database where a plurality of the pattern-based message templates are keys and the corresponding message signatures are values of the key-value database.
5. The method of claim 1 , further comprising the step of decompressing the log file using a decompression index that maps a plurality of message signatures to corresponding pattern-based message templates.
6. The method of claim 5 , wherein the decompression index comprises a key-value database where the message signatures are keys and the corresponding plurality of pattern-based message templates are values of the key-value database.
7. The method of claim 1 , wherein the at least one log message is from one or more historical log messages and a real-time stream of log messages.
8. The method of claim 2 , further comprising reassigning one or more of the message signatures based at least in part on a number of occurrences of each message signature and a number of times each pattern-based message template has occurred across a plurality of the log messages, relative to occurrences of other pattern-based message templates, using the counter.
9. A system, comprising:
a memory; and
at least one processing device, coupled to the memory, operative to implement the following steps:
learning at least one pattern that is repeated across a plurality of log messages in historical log data using a log pattern mining technique that learns the at least one pattern to represent a plurality of historical log messages in the historical log data;
generating a plurality of different pattern-based message templates based at least in part on the learned at least one pattern;
generating a compression index that maps each of the plurality of different pattern-based message templates to a corresponding message signature of a plurality of message signatures; and
performing the following steps, subsequent to the generating the compression index, in response to obtaining at least one log message, wherein the at least one log message comprises at least one pattern-based message template and one or more message variable values, wherein the at least one pattern-based message template comprises at least one portion that is replaced with at least one of the message variable values:
separating the at least one log message into the at least one pattern-based message template and the one or more message variable values using the learned at least one pattern;
obtaining the compression index, generated prior to the obtaining, that maps each of the plurality of different pattern-based message templates to the corresponding message signature of the plurality of message signatures; and
writing the one or more separated message variable values and a message signature, obtained from the compression index, corresponding to the separated at least one pattern-based message template of the at least one log message to a log file.
10. The system of claim 9 , further comprising initializing a counter associated with the at least one pattern-based message template in response to the at least one pattern-based message template not being in the compression index and incrementing the counter associated with the at least one pattern-based message template in response to the at least one pattern-based message template being in the compression index, wherein a given message signature is assigned to a particular pattern-based message template based on a length of the given message signature and a frequency of occurrence of the particular pattern-based message template.
11. The system of claim 10 , further comprising, when a given pattern-based message template is not in the compression index: adding a message signature to the compression index and a decompression index for the given pattern-based message template.
12. The system of claim 9 , further comprising the step of decompressing the log file using a decompression index that maps a plurality of message signatures to corresponding pattern-based message templates.
13. The system of claim 9 , wherein the at least one log message is from one or more historical log messages and a real-time stream of log messages.
14. The system of claim 10 , further comprising reassigning one or more of the message signatures based at least in part on a number of occurrences of each message signature and a number of times each pattern-based message template has occurred across a plurality of the log messages, relative to occurrences of other pattern-based message templates, using the counter.
15. A computer program product, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:
learning at least one pattern that is repeated across a plurality of log messages in historical log data using a log pattern mining technique that learns the at least one pattern to represent a plurality of historical log messages in the historical log data;
generating a plurality of different pattern-based message templates based at least in part on the learned at least one pattern;
generating a compression index that maps each of the plurality of different pattern-based message templates to a corresponding message signature of a plurality of message signatures; and
performing the following steps, subsequent to the generating the compression index, in response to obtaining at least one log message, wherein the at least one log message comprises at least one pattern-based message template and one or more message variable values, wherein the at least one pattern-based message template comprises at least one portion that is replaced with at least one of the message variable values:
separating the at least one log message into the at least one pattern-based message template and the one or more message variable values using the learned at least one pattern;
obtaining the compression index, generated prior to the obtaining, that maps each of the plurality of different pattern-based message templates to the corresponding message signature of the plurality of message signatures; and
writing the one or more separated message variable values and a message signature, obtained from the compression index, corresponding to the separated at least one pattern-based message template of the at least one log message to a log file.
16. The computer program product of claim 15 , further comprising initializing a counter associated with the at least one pattern-based message template in response to the at least one pattern-based message template not being in the compression index and incrementing the counter associated with the at least one pattern-based message template in response to the at least one pattern-based message template being in the compression, wherein a given message signature is assigned to a particular pattern-based message template based on a length of the given message signature and a frequency of occurrence of the particular pattern-based message template.
17. The computer program product of claim 16 , further comprising, when a given pattern-based message template is not in the compression index: adding a message signature to the compression index and a decompression index for the given pattern-based message template.
18. The computer program product of claim 15 , further comprising the step of decompressing the log file using a decompression index that maps a plurality of message signatures to corresponding pattern-based message templates.
19. The computer program product of claim 15 , wherein the at least one log message is from one or more historical log messages and a real-time stream of log messages.
20. The computer program product of claim 16 , further comprising reassigning one or more of the message signatures based at least in part on a number of occurrences of each message signature and a number of times each pattern-based message template has occurred across a plurality of the log messages, relative to occurrences of other pattern-based message templates, using the counter.