Log collection, structuring and processing
The present invention generally relates to log message processing such that events can be detected and alarms can be generated. For example, log messages are generated by a variety of network platforms (e.g., Windows servers, Linux servers, UNIX servers, databases, workstations, etc.). Often, relatively large numbers of logs are generated from these platforms in different formats. A log manager described herein collects such log data using various protocols (e.g., Syslog, SNMP, SMTP, etc.) to determine events. That is, the log manager may communicate with the network platforms using appropriate protocols to collect log messages therefrom. The log manager may then determine events (e.g., unauthorized access, logins, etc.) from the log data and transfer the events to an event manager. The event manager may analyze the events and determine whether alarms should be generated therefrom.
1. A method for use in monitoring one or more platforms of a data system, comprising the steps of:
establishing, on a processing platform, a number of log processing rules for selectively processing logs based on a content of one or more data fields of the said logs, wherein log messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols;
identifying, using said processing platform, a log message associated with a monitored platform;
first operating said processing platform to parse said log message into a number of data fields and determining a field content of at least one of said fields; and
based on said field content, second using said processor to process said log message using said processing rules.
2. A method as set forth in claim 1 , wherein said step of establishing comprises providing at least one rule governing archiving of log messages on a log content dependent basis.
3. A method as set forth in claim 1 , wherein said step of establishing comprises providing at least one rule for distinguishing between first logs and second logs, wherein said first logs are prioritized for review.
4. A method as set forth in claim 3 , wherein said first logs are designated as an event for further processing.
5. A method as set forth in claim 4 , wherein said event relates to one of application monitoring, security, operations and auditing or regulatory compliance.
6. A method as set forth in claim 5 , wherein said step of first operating comprises identifying data fields corresponding to one or more of a log host, a log message source, an IP address, a program and a login.
7. A method as set forth in claim 4 , further comprising promoting said event to an alarm status.
8. A method as set forth in claim 7 , further comprising communicating said alarm status of said event to a device upon the promotion of said event to said alarm status.
9. A method as set forth in claim 4 , further comprising making the event available for viewing on a dashboard.
10. A method as set forth in claim 9 , wherein said dashboard comprises at least one user interface.
11. A method as set forth in claim 10 , wherein said event comprises at least one of said first logs and at least one normalized data field.
12. A method for use in monitoring one or more platforms of a data system, comprising the steps of:
identifying, using a processing platform, a log message to be archived, wherein log messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols;
first operating said processing platform to associate metadata with each of multiple parsed portions of the said identified log message to be archived, wherein said metadata includes information to assist in restoring said message; and
second operating said processing platform to archive said log message together with said metadata in a data structure for enabling restoration of said log message using said metadata.
13. A method for use in processing logs in a data system, comprising the steps of:
providing a computer-based tool for use in accessing archived logs based on metadata describing each of multiple parsed portions of the logs, wherein log messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols, wherein the multiple parsed portions comprise data fields corresponding to one or more of a log host, a log message source, an IP address, a program and a login;
first operating said computer-based tool to receive restoration information associated with one or more fields of said metadata; and
second operating said computer-based tool to restore one or more logs based on said received restoration information.
14. A method for use in monitoring one or more platforms of a data system, comprising the steps of:
establishing an agent protocol defining communications between an agent for acquiring log messages and a processor for processing log messages, wherein log messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols;
providing an agent for monitoring log messages associated with a monitored platform; and
operating said agent to identify a log message of interest, process said log message to parse said message and associate metadata with parsed portions of said message, and provide an output in accordance with said agent protocol, wherein said output includes substantially an entirety of a content of said message together with said metadata.
15. A method for use in monitoring one or more platforms of a data system, comprising the steps of:
establishing an agent protocol defining communications between an agent for acquiring log messages and a processor for processing log messages, wherein log messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols;
receiving, at said processor, a pre-processed log message in accordance with said protocol, where said pre-processed log message includes parsed portions of a raw log message, collectively including substantially an entirety of a content of said raw message, together with metadata defining said parsed portions; and
operating said processor to use said metadata to further process said preprocessed log message.
16. A method for use in processing textual messages in a data system, comprising the steps of:
first using a processor to establish a tagging notation in relation to a subject matter area of a textual message, said tagging notation including a metadata model for describing parsed portions of said textual message, wherein textual messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols; and
second using said processor to establish rules operative for converting said tagging notation into regular expression notation, wherein said tagging notation includes at least a first tag and a second tag wherein said first tag is operative for identifying a parsed portion of said textual message and said second tag is operative to identify a sub rule, wherein said sub rule further defines one of said rules.
17. A method for use in monitoring one or more platforms in a data system, comprising the steps of:
first operating a processing platform to access a Windows event log from a monitored platform; wherein event logs in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols;
second operating said processing platform to identify a timestamp associated with the Windows event log; and
third operating said processing platform to normalize the timestamp such that the time stamp is substantially independent of a processing environment of the monitored platform; wherein said step of third operating comprises accounting for one of a local time zone, a local clock offset, and a local platform time system.