Methods and systems to tag tokens in log messages
This disclosure is directed to tagging tokens or sequences of tokens in log messages generated by a logging source. Event types of log messages in a block of log messages are collected. A series of tagging operations are applied to each log message in the block. For each tagging operation, event types that are qualified to receive the corresponding tag are identified. When a log message is received, the event type is determined and compared with the event types of the block in order to identify a matching event type. The series of tagging operations are applied to the log message to generate a tagged log message with the restriction that each tagging operation only applies a tag to token or sequences of tokens when the event type is qualified to receive the tag. The tagged log message is stored in a data-storage device.
1. An automated method stored in one or more data-storage devices and executed using one or more processors of a computer system for tagging tokens and sequences of tokens in log messages generated by an event source, the method comprising:
determining tags each event type of a block of untagged log messages is qualified to receive, the block of untagged log messages generated by the event source;
performing event-type analysis on a log message generated by the event source to identify an event type of the log message;
determining an event type of the block of untagged log messages that matches the event type of the log message;
applying a series of tagging operations to tokens or sequences of tokens of the log message to generate a tagged log message, each tagging operation applying a tag to a token or a sequence of tokens of the log message when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag; and
storing the tagged log message in a data-storage device.
2. The method of claim 1 wherein determining the tags each event type of untagged log messages in the block of log messages are qualified to receive comprises:
applying event-type analysis to each log message of the block of untagged log messages;
for each event type, determine log messages with the same event type in the block of untagged log messages;
applying the series of tagging operations to each log message in the block of untagged log messages; and
for each tag and for each event type,
determining a count of the log messages with the same event type and with tokens or sequences of tokens that received the same tag;
computing a fraction of log messages with the same event type and the same tag as a ratio of the count of log messages with the same event type that received the tag to the number of log messages with the same event type, and
identifying the event type as qualified to receive the tag when the fraction of log messages with the tag is greater than a tagging threshold.
3. The method of claim 1 wherein determining an event type of the block of untagged log messages that matches the event type of the log message comprises:
comparing the event type of the log message to each of the event types of the block of untagged log message; and
identifying the matching event type of the block of untagged log message as having the same non-parametric tokens as the event type of the log message.
4. The method of claim 1 wherein applying the series of tagging operations to the log message to generate the tagged log message comprises for each tagging operation in the series of tagging operations, applying the tag to tokens or sequences of tokens of the log message, when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag of the tagging operation.
5. A system for tagging tokens and sequences of tokens in log messages generated by an event source, the system comprising:
one or more processors;
one or more data-storage devices; and
machine-readable instructions stored in the one or more data-storage devices that when executed using the one or more processors controls the system to perform the operations comprising:
determining tags each event type of a block of untagged log messages is qualified to receive, the block of untagged log messages generated by the event source;
performing event-type analysis on a log message generated by the event source to identify an event type of the log message;
determining an event type of the untagged block of log messages that matches the event type of the log message;
applying a series of tagging operations to tokens or sequences of tokens of the log message to generate a tagged log message, each tagging operation applying a tag to a token or a sequence of tokens of the log message when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag; and
storing the tagged log message in a data-storage device.
6. The system of claim 5 wherein determining the tags each event type of untagged log messages in the block of log messages is qualified to receive comprises:
applying event-type analysis to each log message of the block of untagged log messages;
for each event type, determine log messages with the same event type in the block of untagged log messages;
applying the series of tagging operations to each log message in the block of untagged log messages; and
for each tag and for each event type,
determining a count of the log messages with the same event type and with tokens or sequences of tokens that received the same tag;
computing a fraction of log messages with the same event type and the same tag as a ratio of the count of log messages with the same event type that received the tag to the number of log messages with the same event type, and
identifying the event type as qualified to receive the tag when the fraction of log messages with the tag is greater than a tagging threshold.
7. The system of claim 5 wherein determining an event type of the block of untagged log messages that matches the event type of the log message comprises:
comparing the event type of the log message to each of the event types of the block of untagged log message; and
identifying the matching event type of the block of untagged log message as having the same non-parametric tokens as the event type of the log message.
8. The system of claim 5 wherein applying the series of tagging operations to the log message to generate the tagged log message comprises for each tagging operation in the series of tagging operations, applying the tag to tokens or sequences of tokens of the log message, when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag of the tagging operation.
9. A non-transitory computer-readable medium encoded with machine-readable instructions that implement a method carried out by one or more processors of a computer system to perform the operations comprising:
determining tags each event type of a block of untagged log messages is qualified to receive, the block of untagged log messages generated by the event source;
performing event-type analysis on a log message generated by the event source to identify an event type of the log message;
determining an event type of the block of untagged log messages that matches the event type of the log message;
applying a series of tagging operations to tokens or sequences of tokens of the log message to generate a tagged log message, each tagging operation applying a tag to a token or a sequence of tokens of the log message when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag; and
storing the tagged log message in a data-storage device.
10. The medium of claim 9 wherein determining the tags each event type of log messages in the block of log messages are qualified to receive comprises:
applying event-type analysis to each log message of the block of untagged log messages;
for each event type, determine log messages with the same event type in the block of untagged log messages;
applying the series of tagging operations to each log message in the block of untagged log messages; and
for each tag and for each event type,
determining a count of the log messages with the same event type and with tokens or sequences of tokens that received the same tag;
computing a fraction of log messages with the same event type and the same tag as a ratio of the count of log messages with the same event type that received the tag to the number of log messages with the same event type, and
identifying the event type as qualified to receive the tag when the fraction of log messages with the tag is greater than a tagging threshold.
11. The medium of claim 9 wherein determining an event type of the block of untagged log messages that matches the event type of the log message comprises:
comparing the event type of the log message to each of the event types of the block of untagged log message; and
identifying the matching event type of the block of untagged log message as having the same non-parametric tokens as the event type of the log message.
12. The medium of claim 9 wherein applying the series of tagging operations to the log message to generate the tagged log message comprises for each tagging operation in the series of tagging operations, applying the tag to tokens or sequences of tokens of the log message, when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag of the tagging operation.