Methods and systems to compound alerts in a distributed computing system
Computational methods and systems described herein are directed to compounding alerts generated in a distributed computing system. A user or system administrator may define a set of multistage process rules that can be used by a log management server application to examine log messages generated by event sources of a multistage process for alerts. A log-message database is searched to identify a log-message file used to record log messages generated by the event sources. A single compound alert indicating that the multistage process rules are satisfied is generated, when log messages of the log-message file that satisfy the rules of the multistage process rules have been identified. Methods may also execute remedial action to correct the multistate process when log messages of the log-message file fail to satisfy at least one rule of the multistage process rules.
1. A process stored in one or more data-storage devices and executed using one or more processors of a computer system to compound alerts of a multistage process executed in a distributed computing system, the process comprising:
searching a log-message database stored in a data-storage device to identify a log-message file generated by event sources of the multistage process;
identifying log messages of the log-message file that satisfy rules of a set of multistage process rules defined for the multistage process; and
generating a compound alert indicating that the set of multistage process rules are satisfied, when log messages of the log-message file that satisfy the rules of the set of multistage process rules have been identified.
2. The process of claim 1 further comprising executing remedial action to correct the multistage process when log messages of the log-message file fail to satisfy at least one rule of the set of multistage process rules.
3. The process of claim 1 wherein identifying log messages of the log-message file that satisfy rules of the set of multistage process rules defined for the multistage process comprises:
for each rule, identifying log messages with time stamps in a time interval associated with the rule; and
to each log message in the time interval,
applying event-type analysis to determine an event type of the log message,
comparing terms of the event type to terms of the rule, and
writing the log message to a buffer when the event type matches the rule.
4. The process of claim 3 further comprising when no log messages in the time interval satisfies the rule, generating an alert that indicates no log message satisfies the rule.
5. A system to compound alerts of a multistage process executed in a distributed computing system, the system comprising:
one or more processors;
one or more data-storage devices; and
machine-readable instructions stored in the one or more data-storage devices that when executed using the one or more processors controls the system to perform operations comprising:
searching a log-message database stored in a data-storage device to identify log-message files generated by event sources of the multistage process;
for each log-message file,
searching the log-message file for a first log message that satisfies a first rule of multistage process rules that confirms completion of a first stage of the multistage process,
when a first log message that satisfies the first rule is identified, incrementing a first stage counter,
searching the log-message file for a second log message that satisfies a second rule of the multistage process rules that confirms completion of a second stage of the multistage process, and
when a second log message that satisfies the second rule is identified, incrementing a second stage counter; and
when the second stage counter is less than the first stage counter, generating a compound alert indicating that not all stages of the multistage process completed successfully.
6. The system of claim 5 further comprising executing remedial action to correct the process, when no log message of the log-message file satisfies the second rule.
7. The system of claim 5 wherein searching the log-message file for the first log message that satisfies the first rule of multistage process rules comprises:
identifying log messages with time stamps in a first stage interval of the first rule; and
to each log message in the first stage interval,
applying event-type analysis to determine an event type of the log messages,
comparing terms of the event type to terms of the first rule, and
writing the log message to a buffer when the event type matches the first rule, when terms of the event type match terms of the first rule.
8. The system of claim 5 wherein searching the log-message file for the second log message that satisfies the second rule of multistage process rules comprises:
identifying log messages with time stamps in a second stage interval of the second rule; and
to each log message in the second stage interval,
applying event-type analysis to determine an event type of the log messages,
comparing terms of the event type to terms of the second rule, and
writing the log message to a buffer when the event type matches the second rule, when terms of the event type match terms of the second rule.
9. A non-transitory computer-readable medium encoded with machine-readable instructions that implement a method carried out by one or more processors of a computer system to perform operations comprising:
searching a log-message database stored in a data-storage device to identify a log-message file of an event source that runs in a computer system of a distributed computing system;
searching the log-message file for a first log message that satisfies a first rule of multistage process rules that confirms completion of a first stage of a multistage process;
when a first log message that satisfies the first rule is identified, searching the log-message file for a second log message that satisfies a second rule of the multistage process rules that confirms completion of a second stage of the multistage process; and
generating a compound alert indicating that the process completed successfully when a log message of the log-message file satisfies the second rule.
10. The medium of claim 9 further comprising executing remedial action to correct the process, when no log message of the log-message file satisfies the second rule.
11. The medium of claim 9 wherein searching the log-message file for the first log message that satisfies the first rule of multistage process rules comprises:
identifying log messages with time stamps in a first stage interval of the first rule; and
to each log message in the first stage interval,
applying event-type analysis to determine an event type of the log messages,
comparing terms of the event type to terms of the first rule, and
writing the log message to a buffer when the event type matches the first rule, when terms of the event type match terms of the first rule.
12. The medium of claim 11 further comprising generating an alert that indicates no log message is in the first stage interval, when no log message of the log-message file satisfies the first rule.
13. The medium of claim 9 wherein searching the log-message file for the second log message that satisfies the second rule of multistage process rules comprises:
identifying log messages with time stamps in a second stage interval of the second rule; and
to each log message in the second stage interval,
applying event-type analysis to determine an event type of the log messages,
comparing terms of the event type to terms of the second rule, and
writing the log message to a buffer when the event type matches the second rule, when terms of the event type match terms of the second rule.
14. The medium of claim 13 further comprising generating an alert that indicates no log message is in the second stage interval, when no log message of the log-message file satisfies the second rule.