Dynamic parsing rules
View Patent ↗Methods, program products, and systems implementing dynamic parsing rules are disclosed. Log data from a variety of log producers can be parsed using parsing rules to generate information about an information system. The parsing rules can include system parsing rules and custom parsing rules. A state machine can be used to detect conflicts between various parsing rules. A central server can distribute the system parsing rules and custom parsing rules to one or more remote servers for distributed processing. In a hierarchical parsing system, a first tier parser can be used to identify types of sources generating the log data. Log data from each type of log source can be sent to a second tier parser that corresponds to the type of log source.
1. A method executed by one or more data processing devices, comprising:
receiving log data;
receiving a first parsing rule and a second parsing rule, the first parsing rule and second parsing rule configured to extract a named value based on the received log data;
identifying a conflict between the first parsing rule and the second parsing rule;
resolving the conflict; and
extracting the named value using a combination of the first parsing rule and the second parsing rule.
2. The method of claim 1 , where receiving the log data includes receiving one or more data messages.
3. The method of claim 1 , where receiving the first parsing rule includes receiving a tag and an expression, the tag including a name corresponding to the value, the expression including a pattern for identifying the value corresponding to the name.
4. The method of claim 1 , where receiving the second parsing rule includes receiving a user-created parsing rule.
5. The method of claim 4 , where receiving the user-created parsing rule includes:
receiving a user selection of a portion of a data message included in the data log;
generating a message pattern based on the user selection; and
designating a user-created tag and the message pattern as the user-created parsing rule.
6. The method of claim 1 , where identifying the conflict between the first parsing rule and the second parsing rule includes:
instantiating one or more state machines based on the first parsing rule and the second parsing rule; and
identifying an end state that is reachable according to both the first parsing rule and the second parsing rule.
7. The method of claim 6 , where at least one of the state machines includes a deterministic finite automaton (DFA).
8. The method of claim 1 , where resolving the conflict includes providing a user interface for modifying the second parsing rule.
9. The method of claim 1 , where extracting the named value includes:
associating a name specified in the first parsing rule or second parsing rule to an identifier; and
storing the identifier in association with a value that corresponds to the name according to the first rule or second rule, the value being exacted from the log data.
10. The method of claim 9 , further comprising:
storing, in a tag catalog, the name and the associated identifier;
receiving a new name for replacing the name; and
associating the new name to the identifier.
11. The method of claim 9 , further comprising:
associating a security setting to a name, the security setting specifying access privileges of a value that corresponds to the name; and
storing the security setting and the name in a tag catalog.
12. The method of claim 11 , where the security setting specifies that the value is to be obfuscated or redacted if accessed by a type of user.
13. The method of claim 12 , where identifying the type of the data message includes identifying a device type of a device or application that generates the data message.
14. The method of claim 1 , further comprising:
creating a first parser based on the first parsing rule and a second parser based on the first parsing rule and the second parsing rule;
identifying a type of a data message included in the data log; and
applying one of the first parser and the second parser to the data message based on the identified type.
15. The method of claim 1 , further comprising distributing the first parsing rule and the second parsing rule to one or more remote data processing units, the remote data processing units configured to receive and process second log data without creating conflicts in parsing rules.
16. A computer program product stored on a storage device, operable to cause one or more processors to perform operations comprising:
receiving log data;
receiving a first parsing rule and a second parsing rule, the first parsing rule and second parsing rule configured to extract a named value based on the received log data;
identifying a conflict between the first parsing rule and the second parsing rule;
resolving the conflict; and
extracting the named value using a combination of the first parsing rule and the second parsing rule.
17. A system comprising:
one or more processors configured to perform operations comprising:
receiving log data;
receiving a first parsing rule and a second parsing rule, the first parsing rule and second parsing rule configured to extract a named value based on the received log data;
identifying a conflict between the first parsing rule and the second parsing rule;
resolving the conflict; and
extracting the named value using a combination of the first parsing rule and the second parsing rule.