Real-time compression of log data
Techniques are provided for compression and decompression of log data. An exemplary method comprises: obtaining a log message, wherein the log message comprises a message template and one or more message variables; obtaining a compression index that maps a plurality of message templates to a corresponding message signature; and writing the one or more message variables and a message signature corresponding to the message template of the log message to a log file. A counter may be maintained for each of a plurality of distinct message templates, and a given message signature may be assigned to a particular message template based on a length of the given message signature and a frequency of occurrence of the particular message template. The compression index comprises, for example, a key/value database where the message templates are keys and the corresponding message signatures are values of the key/value database. A decompression index maps message signatures to corresponding message templates.
1. A method, comprising:
obtaining at least one log message of a plurality of log messages, wherein the at least one log message comprises a message template and one or more message variable values, wherein the message template is repeated across the plurality of the log messages and comprises at least one portion that is replaced with at least one of the message variable values;
obtaining a compression index that maps each of a plurality of different message templates to a corresponding message signature of a plurality of message signatures, wherein a given message signature is assigned to a particular message template based on a length of the given message signature and a frequency of occurrence of the particular message template;
initializing a counter associated with the message template in response to the message template not being in the compression index and incrementing the counter associated with the message template in response to the message template being in the compression index;
adding the message template and the corresponding message signature to the compression index;
obtaining the corresponding message signature from the compression index that corresponds to the message template in the at least one log message;
writing, using at least one processing device, the one or more message variable values and the message signature corresponding to the message template of the at least one log message to a log file; and
reassigning one or more of the message signatures based at least in part on a number of occurrences of each message signature and a number of times each message template has occurred across a plurality of the log messages, relative to occurrences of other message templates, using the counter.
2. The method of claim 1 , further comprising the step of determining whether the message template is in the compression index.
3. The method of claim 2 , further comprising, when a given message template is not in the compression index, adding a message signature to a decompression index for the given message template.
4. The method of claim 1 , wherein the compression index comprises a key/value database where the plurality of message templates are keys and the corresponding message signatures are values of the key/value database.
5. The method of claim 1 , further comprising the step of decompressing the log file using a decompression index that maps a plurality of message signatures to corresponding message templates.
6. The method of claim 5 , wherein the decompression index comprises a key/value database where the message signatures are keys and the corresponding plurality of message templates are values of the key/value database.
7. The method of claim 1 , wherein the message template is not stored in the log file.
8. A system, comprising:
a memory; and
at least one processing device, coupled to the memory, operative to implement the following steps:
obtaining at least one log message of a plurality of log messages, wherein the at least one log message comprises a message template and one or more message variable values, wherein the message template is repeated across the plurality of the log messages and comprises at least one portion that is replaced with at least one of the message variable values;
obtaining a compression index that maps each of a plurality of different message templates to a corresponding message signature of a plurality of message signatures, wherein a given message signature is assigned to a particular message template based on a length of the given message signature and a frequency of occurrence of the particular message template;
initializing a counter associated with the message template in response to the message template not being in the compression index and incrementing the counter associated with the message template in response to the message template being in the compression index;
adding the message template and the corresponding message signature to the compression index;
obtaining the corresponding message signature from the compression index that corresponds to the message template in the at least one log message;
writing, using at least one processing device, the one or more message variable values and the message signature corresponding to the message template of the at least one log message to a log file; and
reassigning one or more of the message signatures based at least in part on a number of occurrences of each message signature and a number of times each message template has occurred across a plurality of the log messages, relative to occurrences of other message templates, using the counter.
9. The system of claim 8 , further comprising, when a given message template is not in the compression index, adding a message signature to a decompression index for the given message template.
10. The system of claim 8 , wherein the compression index comprises a key/value database where the plurality of message templates are keys and the corresponding message signatures are values of the key/value database.
11. The system of claim 8 , further comprising the step of decompressing the log file using a decompression index that maps a plurality of message signatures to corresponding message templates.
12. The system of claim 8 , wherein the message template is not stored in the log file.
13. A computer program product, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:
obtaining at least one log message of a plurality of log messages, wherein the at least one log message comprises a message template and one or more message variable values, wherein the message template is repeated across the plurality of the log messages and comprises at least one portion that is replaced with at least one of the message variable values;
obtaining a compression index that maps each of a plurality of different message templates to a corresponding message signature of a plurality of message signatures, wherein a given message signature is assigned to a particular message template based on a length of the given message signature and a frequency of occurrence of the particular message template;
initializing a counter associated with the message template in response to the message template not being in the compression index and incrementing the counter associated with the message template in response to the message template being in the compression index;
adding the message template and the corresponding message signature to the compression index;
obtaining the corresponding message signature from the compression index that corresponds to the message template in the at least one log message;
writing, using at least one processing device, the one or more message variable values and the message signature corresponding to the message template of the at least one log message to a log file; and
reassigning one or more of the message signatures based at least in part on a number of occurrences of each message signature and a number of times each message template has occurred across a plurality of the log messages, relative to occurrences of other message templates, using the counter.
14. The computer program product of claim 13 , further comprising, when a given message template is not in the compression index, adding a message signature to a decompression index for the given message template.
15. The computer program product of claim 13 , wherein the compression index comprises a key/value database where the plurality of message templates are keys and the corresponding message signatures are values of the key/value database.
16. The computer program product of claim 13 , further comprising the step of decompressing the log file using a decompression index that maps a plurality of message signatures to corresponding message templates.
17. The computer program product of claim 13 , wherein the message template is not stored in the log file.
18. The method of claim 1 , wherein the reassigning is performed using an offline process.
19. The system of claim 8 , wherein the reassigning is performed using an offline process.
20. The computer program product of claim 13 , wherein the reassigning is performed using an offline process.