Efficient event-type-based distributed log-analytics system
The current document is directed to methods and systems that efficiently transmit and process log/event messages within and among distributed computer facilities. By digesting and condensing log/event messages at the message-collector level, the volume of data transmitted from message collectors to message-ingestion-and-processing systems is greatly reduced, which increases system efficiencies by decreasing network overheads and which provides sufficient additional computational bandwidth at the message-collector level to allow message collectors to offload many message-processing tasks from message-ingestion-and-processing system and other downstream processing systems. When the currently disclosed, improved message-collectors carry out message-processing tasks formerly carried out by message-ingestion-and-processing systems and other downstream processing systems, an even greater deduction in the volume of data transmitted from message collectors to message-ingestion-and-processing systems is obtained, further increasing system efficiencies. The decrease in data volume also contributes to increased message-query-processing efficiencies.
1 . An improved log/event-message system, within a distributed computer system, that collects log/event messages from log/event-message sources within the distributed computer system, stores the collected log/event messages, and provides query-based access to the stored log/event-messages, the log/event-message system comprising:
one or more message collectors, incorporated within one or more computer systems, each having one or more processors and one or more memories, which each
receives log/event messages,
compresses the received log/event messages into compressed-message expressions, and
forwards the compressed-message expressions to one or more of one or more message-ingestion-and-processing systems; and
the one or more message-ingestion-and-processing systems, incorporated within one or more computer systems, each having one or more processors and one or more memories, which each
receives compressed-message expressions from one or more of the one or more message collectors, and
stores the received compressed-message expressions in a message database.
2 . The log/event-message system of claim 1 wherein log/event-message sources include:
message-generation-and-reporting components of hardware components of the distributed computer system, including network routers and bridges, network-attached storage devices, and network-interface controllers; and
message-generation-and-reporting components within computer-instruction-implemented components of the distributed computer system, including virtualization layers, operating systems, and applications.
3 . The log/event-message system of claim 1 wherein log/event-messages include text, alphanumeric values, and/or numeric values that represent various types of information, including notification of completed actions, errors, anomalous operating behaviors and conditions, and various types of computational events and warnings.
4 . The log/event-message system of claim 1 wherein the compressed-message expression corresponding to a log/event message includes numeric and/or character-string values corresponding to variable portions of the type of log/event message to which the log/event message belongs.
5 . The log/event-message system of claim 1 wherein the message collector compresses a received log/event message into a compressed-message expression by
applying a first function to the received log/event message to generate an event type for the received log/event message;
using the event type to select an extraction function for the event type;
applying the extraction function to generate a list of 0, 1, or more numeric and/or character-string values; and
combining the event type and list to generate the compressed-message expression.
6 . The log/event-message system of claim 5 wherein the first and extraction functions are:
manually created from templates;
automatically generated by a clustering method; and/or
implemented by a machine-learning system.
7 . The log/event-message system of claim 5 wherein the message collector, after compressing a received log/event message into a compressed-message expression, processes the compressed-message expression by one or more compressed-message-expression processes selected from among:
a filtering process;
an alert process;
a forwarding process;
a sampling process; and
a masking process.
8 . The log/event-message system of claim 7 wherein the filtering process
receives a compressed-message expression;
applies one or more filtering functions and/or rules to the received compressed-message expression or to a modified version of the compressed-message expression, each filtering function and rule either modifying the list or indicating that the received compressed-message expression should be dropped; and
when one of the applied functions or rules indicates that the received compressed-message expression should be dropped, terminates processing of the received compressed-message expression by the message collector, preventing forwarding of the received compressed-message expression to the one or more of one or more message-ingestion-and-processing systems by the message collector.
9 . The log event-message system of claim 7 wherein the alert process
receives a compressed-message expression;
applies one or more alert functions and/or rules to the received compressed-message expression, each alert function and rule indicating whether an alert should be generated and, if an alert should be generated, whether the received compressed-message expression should be dropped;
when one of the applied functions or rules indicates that an alert should be generated,
generates the alert and forwards the alert for transmission; and
when one of the applied functions or rules indicates that the received compressed-message expression should be dropped, terminates processing of the received compressed-message expression by the message collector, preventing forwarding of the received compressed-message expression to the one or more of one or more message-ingestion-and-processing systems by the message collector.
10 . The log/event-message system of claim 7 wherein the forwarding process
receives a compressed-message expression;
applies one or more forwarding functions and/or rules to the received compressed-message expression, each forwarding function and rule indicating whether the received compressed-message expression should be forwarded to a remote computing entity other than one or more of one or more message-ingestion-and-processing systems and, if the received compressed-message expression should be forwarded, whether the received compressed-message expression should be dropped;
when one of the applied functions or rules indicates that received compressed-message expression should be forwarded to a remote computing entity other than one or more of one or more message-ingestion-and-processing systems,
forwards the received compressed-message expression for transmission to the remote computing entity; and
when one of the applied functions or rules indicates that the received compressed-message expression should be dropped, terminates processing of the received compressed-message expression by the message collector, preventing forwarding of the received compressed-message expression to the one or more of one or more message-ingestion-and-processing systems by the message collector.
11 . The log event-message system of claim 7 wherein the sampling process
receives a compressed-message expression;
determines whether the event type included in received compressed-message expression is currently being sampled;
when the event type included in received compressed-message expression is currently being sampled,
determines whether the received compressed-message expression should be dropped, and
when one of the applied functions or rules indicates that the received compressed-message expression should be dropped, terminates processing of the received compressed-message expression by the message collector, preventing forwarding of the received compressed-message expression to the one or more of one or more message-ingestion-and-processing systems by the message collector.
12 . The log/event-message system of claim 7 wherein the masking process
receives a compressed-message expression; and
applies one or more masking functions and/or rules to the received compressed-message expression or to a modified version of the compressed-message expression, each masking function and rule modifying the list to conceal or obscure one or more numeric and/or character-string values in the list.
13 . A method that improves a log/event-message system within a distributed computer system that collects log/event messages from log/event-message sources within the distributed computer system, stores the collected log/event messages, and provides query-based access to the stored log/event-messages, the method comprising:
collecting log/event-messages from the log/event-message sources by one or more message collectors, each message collector
compressing the received log/event messages into compressed-message expressions, and
forwarding the compressed-message expressions to one or more of one or more message-ingestion-and-processing systems; and
receiving compressed-message expressions from one or more of the one or more message collectors by the one or more message-ingestion-and-processing systems, each message-ingestion-and-processing system storing the received compressed-message expressions in a message database.
14 . The method of claim 13 wherein log/event-messages are issued by:
message-generation-and-report in functionality incorporated within hardware components of the distributed computer system, including network routers and bridges, network-attached storage devices, and network-interface controllers; and
message-generation-and-reporting functionality incorporated within computer-instruction-implemented components of the distributed computer system, including virtualization layers, operating systems, and applications.
15 . The method of claim 14 wherein the compressed-message expression corresponding to a log/event message includes numeric and/or character-string values corresponding to variable portions of the type of log/event message to which the log/event message belongs.
16 . The method of claim 14 wherein the message collector compresses a received log/event message into a compressed-message expression by
applying a first function to the received log/event message to generate an event type for the received log/event message;
using the event type to select an extraction function for the event type;
applying the extraction function to generate a list of 0, 1, or more numeric and/or character-string values; and
combining the event type and list to generate the compressed-message expression.
17 . The method of claim 16 wherein the first and extraction functions are:
manually created from templates;
automatically generated by a clustering method;
and/or implemented by a machine-learning system.
18 . The method of claim 17 wherein the message collector, after compressing a received log/event message into a compressed-message expression, processes the compressed-message expression by one or more compressed-message-expression processes selected from among:
a filtering process;
an alert process;
a forwarding process;
a sampling process; and
a masking process.
19 . A physical data-storage device that stores computer instructions that, when executed by processors within computer systems of a log/event-message system within a distributed computer system, control the log/event-message system to:
collect log/event-messages from log event-message sources by one or more message collectors, each message collector
compressing the received log/event messages into compressed-message expressions, and
forwarding the compressed-message expressions to one or more of one or more message-ingestion-and-processing systems; and
receive compressed-message expressions from one or more of the one or more message collectors by the one or more message-ingestion-and-processing systems, each message-ingestion-and-processing system storing the received compressed-message expressions in a message database.