METHOD AND SUBSYSTEM OF A DISTRIBUTED LOG-ANALYTICS SYSTEM THAT AUTOMATICALLY DETERMINE THE SOURCE OF LOG/EVENT MESSAGES
The current document is directed to methods and subsystems within distributed log-analytics systems that automatically and autonomously generate indications of log sources for log/event messages received by the distributed log-analytics systems. The log-source indications can be incorporated in tags associated with received log/event messages to facilitate use of log/event-message information and log/event-message-processing tools contained in content packs provided by designers, manufacturers, and vendors of computational entities by log/event-message systems that collect, process, and store large volumes of log/event messages generated by many different types of computational entities within distributed computer systems. Log-source indications are generated by a combination of using currently available log-source indications associated with log/event messages, event-type-clustering based event-type-to-log source mapping, and machine-learning-based event-type-to-log source mapping.
1 . An improved log/event-message system, within a distributed computer system, that collects log/event messages from log/event-message sources within the distributed computer system, automatically associates log-source tags with those collected log/event messages for which log-source mappings have been learned, stores the collected log/event messages, and provides query-based access to the stored log/event-messages, the log/event-message system comprising:
one or more message collectors, incorporated within one or more computer systems, each having one or more processors and one or more memories, which each
receives log/event messages,
processes the received log/event messages, and
transmits the log/event messages to one or more downstream processing components, including one or more message-ingestion-and-processing systems; and
the one or more message-ingestion-and-processing systems, incorporated within one or more computer systems, each having one or more processors and one or more memories, which each
receives log/event messages from one or more of the one or more message collectors,
processes the received log/event messages, and
transmits the log/event messages to one or more downstream processing components, including a log/event-message query system.
2 . The log/event-message system of claim 1 wherein log/event-message sources include:
message-generation-and-reporting components of hardware components of the distributed computer system, including network routers and bridges, network-attached storage devices, network-interface controllers, and other hardware components and devices; and
message-generation-and-reporting components within computer-instruction-implemented components of the distributed computer system, including virtualization layers, operating systems, and applications running within servers and other types of computer systems.
3 . The log/event-message system of claim 1 wherein log/event-messages include text, alphanumeric values, and/or numeric values that represent various types of information, including notification of completed actions, errors, anomalous operating behaviors and conditions, various types of computational events, warnings, and other such information.
4 . The log/event-message system of claim 1 wherein the log/event-message system automatically associates log-source tags with the collected log/event messages using multiple types of information, the multiple types of information including:
a mapping of log/event messages to event types;
log-source tags associated with log/event messages prior to an initial learning phase;
clustering of event types unmapped to a verified log source;
log/event-message field definitions provided by content packs; and
machine-learned mappings between information derived from event types and log sources.
5 . The log/event-message system of claim 4 wherein the log/event-message system automatically determines an event type for a received log/event message by:
extracting non-variable field values from the received log/event message; and
identifying an event type for which the extracted non-variable fields most closely matches the non-variable fields extracted from the log/event messages of the event type.
6 . The log/event-message system of claim 5 wherein the log/event-message system associates an event type with a group of log/event messages by:
representing each of a set of received log/event messages by a vector generated from the non-variable fields extracted from the log/event messages;
clustering the vectors; and
assigning a unique event type to each cluster of vectors.
7 . The log/event-message system of claim 4 wherein the log-source tags associated with log/event messages prior to an initial learning phase are generated by one of:
human developers, administrators, managers, and other users of the distributed computer system;
one or more external log/event-message systems; and
by configured message collectors and/or message-processing-and-ingestion systems within a controlled test environment.
8 . The log/event-message system of claim 4 wherein the log/event-message system automatically associates a log source with a group of log/event messages by:
clustering event types unmapped to a verified log source by
generating a feature vector for each event type unmapped to a verified log source from an example log/event message of the event type, and
clustering the feature vectors using a distance metric for feature vectors;
assigning a log source to each event-type cluster;
mapping the event types in each cluster to the log source assigned to the cluster; and
mapping log/event messages of the event type to the log source to which the vent type is mapped.
9 . The log/event-message system of claim 8 further comprising:
applying log/event-message field definitions provided by content packs to associate each unmapped event type to a content pack; and
assigning a log source to each event-type cluster by
determining a likely content pack for each event-type cluster, and
assigning to each event-type cluster a log source associated with the determined likely content pack for the event-type cluster.
10 . The log/event-message system of claim 9 wherein a likely content pack for an event-type cluster is a content pack associated with as many or more unmapped event types in the event-type cluster than any other content pack.
11 . The log/event-message system of claim 4 wherein the log/event-message system automatically associates a log source with a group of log/event messages by:
training a machine-learning classifier to map feature vectors, generated from sample log/event messages for event types, to log sources;
inputting a feature vector generated from a sample log/event message for an unmapped event type to the trained machine-learning classifier;
receiving output from the machine-learning classifier; and
when the output indicates that the input feature vector can be assigned to a log source, associating the log source with log/event messages of the event type.
12 . The log/event-message system of claim 11
wherein the output includes a log source and a probability; and
wherein the output indicates that the input feature vector can be assigned to the log source when the probability is greater than a threshold value.
13 . A method that improves a log/event-message system within a distributed computer system that collects log/event messages from log/event-message sources within the distributed computer system, stores the collected log/event messages, and provides query-based access to the stored log/event-messages, the method comprising:
learning mappings, by the log/event-message system, between log sources and event types; and
automatically associating, by the log/event-message system, log-source tags containing log-source indications with those collected log/event messages having event types for which log-source mappings have been learned.
14 . The method system of claim 13 wherein the log/event-message system learns mappings between log sources and event types using multiple types of information, the multiple types of information including:
a mapping of log/event messages to event types;
log-source tags associated with log/event messages prior to an initial learning phase;
clustering of event types unmapped to a verified log source;
log/event-message field definitions provided by content packs; and
machine-learned mappings between information derived from event types and log sources.
15 . The method of claim 14
wherein the log/event-message system automatically determines an event type for a received log/event message by
extracting non-variable field values from the received log/event message, and
identifying an event type for which the extracted non-variable fields most closely matches the non-variable fields extracted from the log/event messages of the event type; and
wherein the log/event-message system associates an event type with a group of log/event messages by
representing each of a set of received log/event messages by a vector generated from the non-variable fields extracted from the log/event messages,
clustering the vectors, and
assigning a unique event type to each cluster of vectors.
16 . The method of claim 14 wherein the log-source tags associated with log/event messages prior to an initial learning phase are generated by one of:
human developers, administrators, managers, and other users of the distributed computer system;
one or more external log/event-message systems; and
by configured message collectors and/or message-processing-and-ingestion systems within a controlled test environment.
17 . The method of claim 14 wherein the log/event-message system automatically associates a log source with a group of log/event messages by:
clustering event types unmapped to a verified log source by
generating a feature vector for each event type unmapped to a verified log source from an example log/event message of the event type, and
clustering the feature vectors using a distance metric for feature vectors;
assigning a log source to each event-type cluster;
mapping the event types in each cluster to the log source assigned to the cluster; and
mapping log/event messages of the event type to the log source to which the vent type is mapped.
18 . The method of claim 17 further comprising:
applying log/event-message field definitions provided by content packs to associate each unmapped event type to a content pack; and
assigning a log source to each event-type cluster by
determining a likely content pack for each event-type cluster, and
assigning to each event-type cluster a log source associated with the determined likely content pack for the event-type cluster, wherein the likely content pack for the event-type cluster is a content pack associated with as many or more unmapped event types in the event-type cluster than any other content pack
19 . The method of claim 18 wherein the log/event-message system automatically associates a log source with a group of log/event messages by:
training a machine-learning classifier to map feature vectors, generated from sample log/event messages for event types, to log sources;
inputting a feature vector generated from a sample log/event message for an unmapped event type to the trained machine-learning classifier;
receiving output from the machine-learning classifier that includes a log source and a probability; and
when the output probability is greater than a threshold value, associating the output log source with log/event messages of the event type.
20 . A physical data-storage device that stores computer instructions that, when executed by processors within computer systems of a log/event-message system within a distributed computer system, control the log/event-message system to:
collect log/event messages from log/event-message sources within the distributed computer system;
learn a mapping between log sources and event types;
automatically associating log-source tags containing log-source indications with those collected log/event messages having event types for which log-source mappings have been learned:
store the collected log/event messages; and
provide query-based access to the stored log/event-messages.