System, method, and computer program for automatic parser creation
The present disclosure describes a system, method, and computer program for automatically creating a parser for a log group. A parser-creation system groups logs that do not satisfy conditions for an existing parser, enables a user to select a log group for parser creation, and automatically creates a parser for the selected log group. In creating a parser, the system extracts values and keys value pairs from the log group and identifies the corresponding normalized output fields and regular expressions for the values and key-value pairs. To identify normalized fields corresponding to values and key-value pairs, the system compares the values and key-value pairs to one or more knowledgebases that include: (1) regular expressions from existing parsers, (2) regular expressions for value types associated with normalized fields, and (3) a list of keys in key-value pairs associated with normalized fields. As the system learns new token-to-normalized fields relationships, the system adds the relationships to its knowledgebase.
1. A method, performed by a computer system, for creating a parser for a log group that parses and normalizes log data, the method comprising:
loading a plurality of logs into the system;
for each log in the plurality of logs, determining whether the log satisfies conditions for an existing parser;
in response to a log satisfying conditions for an existing parser, associating the log with said existing parser;
tokenizing logs that do not satisfy conditions for an existing parser;
grouping the logs that do not satisfy conditions for an existing parser to create one or more log groups, wherein the logs are grouped based on token patterns;
displaying the log groups in a first user interface and enabling a user to select a log group for parser creation;
receiving a user selection of a log group for parser creation;
enabling the user to select a vendor and an event type for the selected log group;
receiving the user selection of a vendor and event type for the selected log group;
creating a parser for the selected log group, wherein creating the parser comprises:
creating conditions for the parser based on literals common to each log in the log group;
identifying any required normalized fields for the parser based on the event type;
identifying tokens for the selected log group;
identifying a plurality of normalized fields that correspond to the tokens in the selected log group, wherein identifying a plurality of normalized fields comprises:
comparing each of the tokens to a first knowledgebase of regular expressions associated with normalized fields based on existing parsers accessible to the system,
in response to a token satisfying one of the regular expressions in the first knowledgebase, determining that the token corresponds to the normalized field associated with the satisfied regular expression,
in response to the token not satisfying one of the regular expressions in the first knowledgebase, determining whether the token includes a key in a second knowledgebase of known key names associated with normalized fields used by the system, and, wherein for at least one token that does not satisfy one of the regular expressions in the first knowledgebase, determining whether the token includes a value that satisfies a regular expression for a value type in a third knowledgebase of regular expressions for value types associated with normalized fields used by the system, wherein the first knowledgebase, second knowledgebase, and third knowledgebase are separate libraries, and
in response to the token including a key or a value type associated with a normalized field used by the system, determining that the token corresponds to said normalized field;
mapping each of the identified normalized fields to a regular expression and an example token from the log group;
providing a second user interface that enables a user to view the mapping of identified normalized fields to regular expressions and example tokens, wherein the user interface includes an indication of which of the identified normalized fields are required fields for the parser and wherein the user interface also displays any required normalized field not mapped to a regular expression and an example token;
enabling a user to modify and add to the mapping of identified normalized fields to regular expressions and example tokens; and
creating a parser for the log group based on the mapping, wherein the parser is associated with the vendor and event type selected by the user, as well as the conditions for the log group.
2. The method of claim 1 , wherein mapping an identified normalized field to a regular expression comprises the following:
in response to the identified normalized field corresponding to a token that satisfies one of the regular expressions in the first knowledgebase, mapping the identified normalized field to the regular expression satisfied by the token;
in response to the identified normalized field corresponding to a token that satisfies one of the regular expressions in the third knowledgebase, mapping the identified normalized field to the regular expression satisfied by the token and adding the mapping of the identified normalized field to said regular expression to the first knowledgebase;
in response to the identified normalized field corresponding to a token that is not satisfied by a regular expression in the first or third knowledgebase, but includes a key associated with a normalized field used by the system, creating a regular expression for the corresponding token, mapping the identified normalized field to the created regular expression, and adding the mapping of the identified normalized field to the created regular expression to the first knowledge base of known regular expressions.
3. The method of claim 1 , wherein identifying required normalized fields for the parser comprises the following:
identifying pre-existing parsers accessible to the system for the same event type as the event type selected by the user for the log group;
obtaining a superset of normalized fields extracted in all the pre-existing parsers for the same event type; and
identifying the normalized fields within the superset that are common to all the preexisting parsers for the same event type as the required normalized fields for the parser.
4. A non-transitory computer-readable medium comprising a computer program that, when executed by a computer system, enables the computer system to perform the following method for creating a parser for a log group that parses and normalizes log data, the method comprising:
loading a plurality of logs into the system;
for each log in the plurality of logs, determining whether the log satisfies conditions for an existing parser;
in response to a log satisfying conditions for an existing parser, associating the log with said existing parser;
tokenizing logs that do not satisfy conditions for an existing parser;
grouping the logs that do not satisfy conditions for an existing parser to create one or more log groups, wherein the logs are grouped based on token patterns;
displaying the log groups in a first user interface and enabling a user to select a log group for parser creation;
receiving a user selection of a log group for parser creation;
enabling the user to select a vendor and an event type for the selected log group;
receiving the user selection of a vendor and event type for the selected log group;
creating a parser for the selected log group, wherein creating the parser comprises:
creating conditions for the parser based on literals common to each log in the log group;
identifying any required normalized fields for the parser based on the event type;
identifying tokens for the selected log group;
identifying a plurality of normalized fields that correspond to the tokens in the selected log group, wherein identifying a plurality of normalized fields comprises:
comparing each of the tokens to a first knowledgebase of regular expressions associated with normalized fields based on existing parsers accessible to the system,
in response to a token satisfying one of the regular expressions in the first knowledgebase, determining that the token corresponds to the normalized field associated with the satisfied regular expression,
in response to the token not satisfying one of the regular expressions in the first knowledgebase, determining whether the token includes a key in a second knowledgebase of known key names associated with normalized fields used by the system, and, wherein for at least one token that does not satisfy one of the regular expressions in the first knowledgebase, determining whether the token includes a value that satisfies a regular expression for a value type in a third knowledgebase of regular expressions for value types associated with normalized fields used by the system, wherein the first knowledgebase, second knowledgebase, and third knowledgebase are separate libraries, and
in response to the token including a key or a value type associated with a normalized field used by the system, determining that the token corresponds to said normalized field;
mapping each of the identified normalized fields to a regular expression and an example token from the log group;
providing a second user interface that enables a user to view the mapping of identified normalized fields to regular expressions and example tokens, wherein the user interface includes an indication of which of the identified normalized fields are required fields for the parser and wherein the user interface also displays any required normalized field not mapped to a regular expression and an example token;
enabling a user to modify and add to the mapping of identified normalized fields to regular expressions and example tokens; and
creating a parser for the log group based on the mapping, wherein the parser is associated with the vendor and event type selected by the user, as well as the conditions for the log group.
5. The non-transitory computer-readable medium of claim 4 , wherein mapping an identified normalized field to a regular expression comprises the following:
in response to the identified normalized field corresponding to a token that satisfies one of the regular expressions in the first knowledgebase, mapping the identified normalized field to the regular expression satisfied by the token;
in response to the identified normalized field corresponding to a token that satisfies one of the regular expressions in the third knowledgebase, mapping the identified normalized field to the regular expression satisfied by the token and adding the mapping of the identified normalized field to said regular expression to the first knowledgebase;
in response to the identified normalized field corresponding to a token that is not satisfied by a regular expression in the first or third knowledgebase, but includes a key associated with a normalized field used by the system, creating a regular expression for the corresponding token, mapping the identified normalized field to the created regular expression, and adding the mapping of the identified normalized field to the created regular expression to the first knowledge base of known regular expressions.
6. The non-transitory computer-readable medium of claim 4 , wherein identifying required normalized fields for the parser comprises the following:
identifying pre-existing parsers accessible to the system for the same event type as the event type selected by the user for the log group;
obtaining a superset of normalized fields extracted in all the pre-existing parsers for the same event type; and
identifying the normalized fields within the superset that are common to all the preexisting parsers for the same event type as the required normalized fields for the parser.
7. A computer system for creating a parser for a log group that parses and normalizes log data, the system comprising:
one or more processors;
one or more physical memory units coupled to the one or more processors, wherein the one or more memory units store instructions that, when executed by the one or more processors, cause the system to perform the operations of:
loading a plurality of logs into the system;
for each log in the plurality of logs, determining whether the log satisfies conditions for an existing parser;
in response to a log satisfying conditions for an existing parser, associating the log with said existing parser;
tokenizing logs that do not satisfy conditions for an existing parser;
grouping the logs that do not satisfy conditions for an existing parser to create one or more log groups, wherein the logs are grouped based on token patterns;
displaying the log groups in a first user interface and enabling a user to select a log group for parser creation;
receiving a user selection of a log group for parser creation;
enabling the user to select a vendor and an event type for the selected log group;
receiving the user selection of a vendor and event type for the selected log group;
creating a parser for the selected log group, wherein creating the parser comprises:
creating conditions for the parser based on literals common to each log in the log group;
identifying any required normalized fields for the parser based on the event type;
identifying tokens for the selected log group;
identifying a plurality of normalized fields that correspond to the tokens in the selected log group, wherein identifying a plurality of normalized fields comprises:
comparing each of the tokens to a first knowledgebase of regular expressions associated with normalized fields based on existing parsers accessible to the system,
in response to a token satisfying one of the regular expressions in the first knowledgebase, determining that the token corresponds to the normalized field associated with the satisfied regular expression,
in response to the token not satisfying one of the regular expressions in the first knowledgebase, determining whether the token includes a key in a second knowledgebase of known key names associated with normalized fields used by the system, and, wherein for at least one token that does not satisfy one of the regular expressions in the first knowledgebase, determining whether the token includes a value that satisfies a regular expression for a value type in a third knowledgebase of regular expressions for value types associated with normalized fields used by the system, wherein the first knowledgebase, second knowledgebase, and third knowledgebase are separate libraries, and
in response to the token including a key or a value type associated with a normalized field used by the system, determining that the token corresponds to said normalized field;
mapping each of the identified normalized fields to a regular expression and an example token from the log group;
identifying any required normalized fields for the parser based on the event type;
providing a second user interface that enables a user to view the mapping of identified normalized fields to regular expressions and example tokens, wherein the user interface includes an indication of which of the identified normalized fields are required fields for the parser and wherein the user interface also displays any required normalized field not mapped to a regular expression and an example token;
enabling a user to modify and add to the mapping of identified normalized fields to regular expressions and example tokens; and
creating a parser for the log group based on the mapping, wherein the parser is associated with the vendor and event type selected by the user, as well as the conditions for the log group.
8. The system of claim 7 , wherein mapping an identified normalized field to a regular expression comprises the following:
in response to the identified normalized field corresponding to a token that satisfies one of the regular expressions in the first knowledgebase, mapping the identified normalized field to the regular expression satisfied by the token;
in response to the identified normalized field corresponding to a token that satisfies one of the regular expressions in the third knowledgebase, mapping the identified normalized field to the regular expression satisfied by the token and adding the mapping of the identified normalized field to said regular expression to the first knowledgebase;
in response to the identified normalized field corresponding to a token that is not satisfied by a regular expression in the first or third knowledgebase, but includes a key associated with a normalized field used by the system, creating a regular expression for the corresponding token, mapping the identified normalized field to the created regular expression, and adding the mapping of the identified normalized field to the created regular expression to the first knowledge base of known regular expressions.
9. The system of claim 7 , wherein identifying required normalized fields for the parser comprises the following:
identifying pre-existing parsers accessible to the system for the same event type as the event type selected by the user for the log group;
obtaining a superset of normalized fields extracted in all the pre-existing parsers for the same event type; and
identifying the normalized fields within the superset that are common to all the preexisting parsers for the same event type as the required normalized fields for the parser.