Systems and methods for trie-based automated discovery of patterns in computer logs
Systems and methods for tokenization of log records for efficient data storage, log querying, and log data analytics can utilize a trie pattern conversion of the log files, storing trie data pattern IDs, free parameters, and metadata instead of the entire log record. New trie patterns can be discovered automatically by counting the occurrences of tokens matching wildcards for existing patterns.
1. A system, comprising:
a processor configured to:
collect logs from a plurality of applications;
tokenize a log into a sequence of tokens, each token being a character string;
based at least in part on the sequence of tokens, match the log to a pattern represented by a sequence of nodes stored in a trie, the pattern being associated with a unique pattern identifier, wherein at least one node in the sequence of nodes is a wildcard node configured to match any subsequence of tokens, another node matches one of the tokens in the sequence of tokens, and one special expression node in the sequence of nodes is configured to match any sequence of tokens based on a predefined expression;
extract a set of free parameters and a set of metadata from the log, each free parameter comprising a subsequence of tokens, one of the free parameters from the set of free parameters matching the wildcard node, and another parameter from the set of free parameters matching the special expression node;
store the log as a combination of the unique pattern identifier, the set of free parameters, and the set of metadata, the stored combination comprising enough information to recreate the log, wherein the unique pattern identifier is associated with a last matched node in the sequence of nodes; and
perform a query at least in part by tokenizing a query request and matching the tokenized query request to one or more patterns stored in the trie; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system recited in claim 1 , wherein the matching comprises:
matching iteratively, token by token, the sequence of tokens to the sequence of nodes stored in the trie.
3. The system recited in claim 1 , wherein the sequence of tokens comprises character strings delimited by spaces.
4. A method, comprising:
collecting logs from a plurality of applications;
tokenizing a log into a sequence of tokens, each token being a character string;
based at least in part on the sequence of tokens, matching the log to a pattern represented by a sequence of nodes stored in a trie, the pattern being associated with a unique pattern identifier, wherein at least one node in the sequence of nodes is a wildcard node configured to match any subsequence of tokens, another node matches one of the tokens in the sequence of tokens, and one special expression node in the sequence of nodes is configured to match any sequence of tokens based on a predefined expression;
extracting a set of free parameters and a set of metadata from the log, each free parameter comprising a subsequence of tokens, one of free parameters from the set of free parameters matching the wildcard node, and another parameter from the set of free parameters matching the special expression node;
storing the log as a combination of the unique pattern identifier, the set of free parameters, and the set of metadata, the stored combination comprising enough information to recreate the log, wherein the unique pattern identifier is associated with a last matched node in the sequence of nodes; and
performing a query at least in part by tokenizing a query request and matching the tokenized query request to one or more patterns stored in the trie.
5. The method of claim 4 , wherein the matching comprises:
matching iteratively, token by token, the sequence of tokens to the sequence of nodes stored in the trie.
6. The method of claim 4 , wherein the sequence of tokens comprises character strings delimited by spaces.
7. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
collecting logs from a plurality of applications;
tokenizing a log into a sequence of tokens, each token being a character string;
based at least in part on the sequence of tokens, matching the log to a pattern represented by a sequence of nodes stored in a trie, the pattern being associated with a unique pattern identifier, wherein at least one node in the sequence of nodes is a wildcard node configured to match any subsequence of tokens, another node matches one of the tokens in the sequence of tokens, and one special expression node in the sequence of nodes is configured to match any sequence of tokens based on a predefined expression;
extracting a set of free parameters and a set of metadata from the log, each free parameter comprising a subsequence of tokens, one of free parameters from the set of free parameters matching the wildcard node, and one other from the set of free parameters matching the special expression node;
storing the log as a combination of the unique pattern identifier, the set of free parameters, and the set of metadata, the stored combination comprising enough information to recreate the log, wherein the unique pattern identifier is associated with a last matched node in the sequence of nodes; and
performing a query at least in part by tokenizing a query request and matching the tokenized query request to one or more patterns stored in the trie.
8. The computer program product of claim 7 , wherein the matching comprises:
matching iteratively, token by token, the sequence of tokens to the sequence of nodes stored in the trie.
9. The computer program product of claim 7 , wherein the sequence of tokens comprises character strings delimited by spaces.
10. A system, comprising:
a processor configured to:
collect logs from a plurality of applications;
tokenize a log into a sequence of tokens, each token being a character string;
based at least in part on the sequence of tokens, match the log to a plurality of patterns stored in a trie, each pattern from the plurality of patterns being associated with a unique pattern identifier, each pattern being represented by a respective sequence of nodes, wherein at least one node in the respective sequence of nodes is a wildcard node configured to match any subsequence of tokens, and another node matches one of the tokens in the sequence of tokens;
extract, from the log, a set of metadata and a set of free parameters for each pattern from the plurality of patterns, each free parameter comprising a subsequence of tokens, one of the free parameters from the set of free parameters matching a respective wildcard node;
select a best pattern of the plurality of patterns, the best pattern being selected to be the pattern with a smallest extracted set of free parameters, the best pattern being associated with a best pattern identifier;
store the log as a combination of the best pattern identifier, a respective set of free parameters associated with the best pattern, and the set of metadata, the stored combination comprising enough information to recreate the log, wherein the best pattern identifier is associated with a last matched node in the sequence of nodes;
perform a query at least in part by tokenizing a query request and matching the tokenized query request to one or more patterns stored in the trie; and
a memory coupled to the processor and configured to provide the processor with instructions.
11. The method of claim 4 , further comprising:
matching the log to a plurality of patterns stored in the trie, each pattern in the plurality of
patterns being associated with a respective set of free parameters, the respective set of free parameters being extracted from the log based on matching the log to the pattern;
selecting a best pattern of the plurality of patterns, the best pattern being selected to be the
pattern with a smallest associated set of free parameters, the best pattern being associated with a best pattern identifier; and
storing the log as a combination of the best pattern identifier, a respective set of free
parameters associated with the best pattern, and the set of metadata, the stored combination comprising enough information to recreate the log.