Efficient event-type-based log/event-message processing in a distributed log-analytics system
The current document is directed to methods and systems that efficiently process log/event messages within and among distributed computer facilities. Various different types of initial processing steps may be applied to a stream of log/event messages received by a message-collector system or a message-ingestion-and-processing system. By including a pre-processing step two identify the type of a received log/event message, and by specifying initial-processing-step criteria with respect to log/event-message types, significant increases in the efficiency of log/event-message preprocessing by message-collector systems and message-ingestion-and-processing systems is achieved.
1. An improved log/event-message system, within a distributed computer system, that collects log/event messages from log/event-message sources within the distributed computer system, stores the collected log/event messages, and provides query-based access to the stored log/event-messages, the log/event-message system comprising:
one or more message collectors, incorporated within one or more computer systems, each having one or more processors and one or more memories, which each
receives log/event messages,
processes the received log/event messages, and
transmits the log/event messages to one or more downstream processing components, including one or more message-ingestion-and-processing systems; and
the one or more message-ingestion-and-processing systems, incorporated within one or more computer systems, each having one or more processors and one or more memories, which each
receives log/event messages from one or more of the one or more message collectors,
processes the received log/event messages, and
transmits the log/event messages to one or more downstream processing components, including a log/event-message query system,
one or more of the one or more message collectors and the one or more message-ingestion-and-processing systems processing the received log/event messages by applying message-processing rules that contain criteria vectors to the received log/event messages.
2. The log/event-message system of claim 1 wherein log/event-message sources include:
message-generation-and-reporting components of hardware components of the distributed computer system, including network routers and bridges, network-attached storage devices, network-interface controllers, and other hardware components and devices; and
message-generation-and-reporting components within computer-instruction-implemented components of the distributed computer system, including virtualization layers, operating systems, and applications running within servers and other types of computer systems.
3. The log/event-message system of claim 1 wherein log/event-messages include text, alphanumeric values, and/or numeric values that represent various types of information, including notification of completed actions, errors, anomalous operating behaviors and conditions, various types of computational events, warnings, and other such information.
4. The log/event-message system of claim 1 wherein message-processing rules that contain criteria vectors each comprises:
a criteria portion that consists of one of
a single criteria vector, and
multiple criteria vectors;
an action portion; and
a parameter portion.
5. The log/event-message system of claim 4 wherein the criteria portion of a message-processing rule is evaluated with respect to a received log/event message to determine whether the message-processing rule is applicable to the received log/event message.
6. The log/event-message system of claim 4 wherein the action portion of a message-processing rule specifies a log/event-message-processing action carried out when the message-processing rule is applied to a log/event-message-processing action.
7. The log/event-message system of claim 4 wherein the parameter portion of a message-processing rule includes additional information needed by a message collector and/or a message-ingestion-and-processing system to carry out the log/event-message-processing action specified by the action portion of the message-processing rule.
8. The log/event-message system of claim 4 wherein a criteria vector is associated with a log/event-message type and includes a number of elements N+1, where N is the number of variable fields within log/event messages of the log/event-message type.
9. The log/event-message system of claim 8 wherein the first element of the criteria vector indicates the log/event-message type.
10. The log/event-message system of claim 8 wherein the second through N th elements of the criteria vector each contains either a null value or a field expression.
11. The log/event-message system of claim 10 wherein a field expression in the n th element of a criteria vector comprises one or more sub-expressions.
12. The log/event-message system of claim 11 wherein a sub-expression comprises one of:
a value representing a possible value of the n th log/message field in log/event messages of the log/event-message type associated with the criteria vector and the operator a relational-comparison operator;
a value/operator pair, the value representing a possible value of the n th log/message field in log/event messages of the log/event-message type associated with the criteria vector and the operator a relational-comparison operator;
a sub-expression containing two or more values, value/operator pairs, or sub-expressions logically joined by OR Boolean operators; and
a sub-expression containing two or more values, value/operator pairs, or sub-expressions logically joined by AND Boolean operators.
13. The log/event-message system of claim 4 wherein the criteria portion of a message-processing rule is evaluated with respect to a received log/event message by:
determining a log/event-message type to which the log/event message belongs;
when a criteria vector contained in the criteria portion of the message-processing rule evaluates to FALSE when evaluated with respect to the received log/event message or when no criteria vector contained in the criteria portion of the message-processing rule includes log/event-message type, determining that the message-processing rule is not applicable to the received log/event message; and
otherwise determining that the message-processing rule is applicable to the received log/event message.
14. The log/event-message system of claim 13 wherein a criteria vector is evaluated with respect to the received log/event message by:
when a field expression contained in any element of the criteria vector other than the first element of the criteria vector evaluates to FALSE with respect to the corresponding field of the received log/event message, determining that the criteria vector evaluates to FALSE with respect to the log/event message; and
otherwise determining that the criteria vector evaluates to TRUE with respect to the received log/event message.
15. The log/event-message system of claim 14 wherein a field expression evaluates to TRUE when no sub-expression within the field expression evaluates to FALSE.
16. The log/event-message system of claim 15 wherein a sub-expression within a field expression is evaluated with respect to a corresponding field of the received log/event message by:
when the sub-expression is a value representing a possible value of the corresponding field,
determining that the sub-expression within the field expression evaluates to TRUE when the corresponding field of the received log/event message is equal to the sub-expression value, and
otherwise determining that the sub-expression within the field expression evaluates to FALSE;
when the sub-expression is a value/operator pair,
determining that the sub-expression within the field expression evaluates to TRUE when a Boolean expression constructed from corresponding field of the received log/event message followed by the operator of the value/operator pair followed by the value of the value/operator pair is TRUE, and
otherwise determining that the sub-expression within the field expression evaluates to FALSE;
when the sub-expression is a sub-expression containing two or more values, value/operator pairs, or sub-expressions logically joined by OR Boolean operators,
determining that the sub-expression within the field expression evaluates to TRUE when any of the two or more values, value/operator pairs, or sub-expressions evaluates to TRUE, and
otherwise determining that the sub-expression within the field expression evaluates to FALSE; and
when the sub-expression is a sub-expression containing two or more values, value/operator pairs, or sub-expressions logically joined by AND Boolean operators,
determining that the sub-expression within the field expression evaluates to TRUE when all of the two or more values, value/operator pairs, or sub-expressions evaluates to TRUE, and
otherwise determining that the sub-expression within the field expression evaluates to FALSE.
17. A method that improves a log/event-message system within a distributed computer system that collects log/event messages from log/event-message sources within the distributed computer system, stores the collected log/event messages, and provides query-based access to the stored log/event-messages, the method comprising:
processing received log/event messages, by one or more message collectors that are each incorporated within one or more computer systems having one or more processors and one or more memories and that each receives log/event messages and transmits the log/event messages to one or more downstream processing components, including one or more message-ingestion-and-processing systems, by applying message-processing rules that contain criteria vectors to the received log/event messages; and/or
processing received log/event messages, by one or more of the message-ingestion-and-processing systems that are each incorporated within one or more computer systems having one or more processors and one or more memories and that each receives log/event messages and transmits the log/event messages to one or more downstream processing components, including, including a log/event-message query system, by applying message-processing rules that contain criteria vectors to the received log/event messages.
18. The method of claim 17
wherein message-processing rules that contain criteria vectors each comprises
a criteria portion that consists of one of
a single criteria vector, and
multiple criteria vectors,
an action portion, and
a parameter portion;
wherein the criteria portion of a message-processing rule is evaluated with respect to a received log/event message to determine whether the message-processing rule is applicable to the received log/event message;
wherein the action portion of a message-processing rule specifies a log/event-message-processing action carried out when the message-processing rule is applied to a log/event-message-processing action; and
wherein the parameter portion of a message-processing rule includes additional information needed by a message collector and/or a message-ingestion-and-processing system to carry out the log/event-message-processing action specified by the action portion of the message-processing rule.
19. The method of claim 18
wherein a criteria vector is associated with a log/event-message type and includes a number of elements N+1, where N is the number of variable fields within log/event messages of the log/event-message type;
wherein the first element of the criteria vector indicates the log/event-message type;
wherein the second through N th elements of the criteria vector each contains either a null value or a field expression;
wherein a field expression in the n th element of a criteria vector comprises one or more sub-expressions; and
wherein a sub-expression comprises one of
a value representing a possible value of the n th log/message field in log/event messages of the log/event-message type associated with the criteria vector and the operator a relational-comparison operator,
a value/operator pair, the value representing a possible value of the n th log/message field in log/event messages of the log/event-message type associated with the criteria vector and the operator a relational-comparison operator,
a sub-expression containing two or more values, value/operator pairs, or sub-expressions logically joined by OR Boolean operators, and
a sub-expression containing two or more values, value/operator pairs, or sub-expressions logically joined by AND Boolean operators,
message-generation-and-reporting functionality incorporated within hardware components of the distributed computer system, including network routers and bridges, network-attached storage devices, network-interface controllers, and other hardware components and devices; and
message-generation-and-reporting functionality incorporated within computer-instruction-implemented components of the distributed computer system, including virtualization layers, operating systems, and applications running within servers and other types of computer systems.
20. A physical data-storage device that stores computer instructions that, when executed by processors within computer systems of a log/event-message system within a distributed computer system, control the log/event-message system to:
process received log/event messages, by one or more message collectors that are each incorporated within one or more computer systems having one or more processors and one or more memories and that each receives log/event messages and transmits the log/event messages to one or more downstream processing components, including one or more message-ingestion-and-processing systems, by applying message-processing rules that contain criteria vectors to the received log/event messages; and/or
process received log/event messages, by one or more of the message-ingestion-and-processing systems that are each incorporated within one or more computer systems having one or more processors and one or more memories and that each receives log/event messages and transmits the log/event messages to one or more downstream processing components, including, including a log/event-message query system, by applying message-processing rules that contain criteria vectors to the received log/event messages.