Compression, searching, and decompression of log messages
Log messages are compressed, searched, and decompressed. A dictionary is used to store non-numeric expressions found in log messages. Both numeric and non-numeric expressions found in log messages are represented by placeholders in a string of log “type” information. Another dictionary is used to store the log type information. A compressed log message contains a key to the log-type dictionary and a sequence of values that are keys to the non-numeric dictionary and/or numeric values. Searching may be performed by parsing a search query into subqueries that target the dictionaries and/or content of the compressed log messages. A dictionary may reference segments that contain a number of log messages, so that all log message need not be considered for some searches.
1. A non-transitory computer-readable medium comprising instructions to:
receive a query to search a plurality of compressed log messages;
generate subqueries based on the query, each of the subqueries being:
a non-numeric expression subquery that is executable on a non-numeric expression dictionary that stores non-numeric expressions of variables;
a log-type subquery that is executable on a log-type dictionary that stores log information that excludes expressions of variables and numeric values; or
a numeric subquery that is executable on the plurality of compressed log messages that reference the log-type dictionary;
execute the subqueries; and
output a collection of results of the execution of the subqueries.
2. The non-transitory computer-readable medium of claim 1 , wherein the query comprises a search phrase, and wherein the instructions are further to generate multiple different subqueries from the search phrase.
3. The non-transitory computer-readable medium of claim 1 , wherein the instructions are to execute a numeric subquery by:
identifying, in the log-type dictionary, log-type keys that correspond to log information that contains numeric placeholders;
identifying a subset of the plurality of compressed log messages that contain the log-type keys; and
comparing a value of the numeric subquery to values stored in the subset of the plurality of compressed log messages.
4. The non-transitory computer-readable medium of claim 1 , wherein the plurality of log messages is divided into a plurality of segments;
the non-numeric expression dictionary correlates the non-numeric expressions of variables to the plurality of segments;
the log-type dictionary correlates the log information to the plurality of segments; and
the instructions are further to match a subquery to a non-numeric expression in the non-numeric expression dictionary or a log-type in the log-type dictionary and limit further processing of the subquery to a subset of the plurality of segments correlated to the non-numeric expression or the log-type.
5. The non-transitory computer-readable medium of claim 1 , wherein the instructions are further to combine the subqueries using a logical operator.
6. The non-transitory computer-readable medium of claim 1 , wherein the instructions are further to generate different subqueries based on different interpretations of a token containing a wildcard.