Log collection, structuring and processing
The present invention generally relates to log message processing such that events can be detected and alarms can be generated. For example, log messages are generated by a variety of network platforms (e.g., Windows servers, Linux servers, UNIX servers, databases, workstations, etc.). Often, relatively large numbers of logs are generated from these platforms in different formats. A log manager described herein collects such log data using various protocols (e.g., Syslog, SNMP, SMTP, etc.) to determine events. That is, the log manager may communicate with the network platforms using appropriate protocols to collect log messages therefrom. The log manager may then determine events (e.g., unauthorized access, logins, etc.) from the log data and transfer the events to an event manager. The event manager may analyze the events and determine whether alarms should be generated therefrom.
1. A computer-implemented method of determining a classification of a log message generated by one or more platforms of a data system, comprising the steps of:
receiving, at a log agent, a log message from the one or more platforms of the data system;
parsing an identifier out of the log message;
attempting to match the identifier to a known classification that represents a type of device or process responsible for generating the log message;
assigning the classification to the log message responsive to a successful match during the attempting; and
transmitting to a log manager, a request for information related to a known classification for the identifier responsive to an unsuccessful match during the attempting; and
responsive to receiving a failed known classification message from the log manager;
creating a new classification; and
assigning the identifier to the new classification.
2. The method of claim 1 , wherein after the assigning responsive to a successful match during the attempting, the method further comprises:
sending the log message to a log manager for processing.
3. The method of claim 1 , further comprising:
receiving, at the log agent, the known classification from the log manager.
4. The method of claim 3 , further comprising:
adding, at the log agent, the identifier to a lookup table that is mapped to the received known classification.
5. The method of claim 3 , further comprising:
assigning, at the log agent, all received log messages to the returned known classification when the identifier is parsed out of the received log messages.
6. The method of claim 1 , further comprising:
queuing the log message to an unprocessed log queue responsive an unsuccessful match during the attempting.
7. The method of claim 1 , wherein the identifier comprises at least one of an Internet Protocol (IP) address, a hostname, and a domain name.
8. The method of claim 1 , wherein the identifier comprises an IP address, and the method further comprises after the parsing step:
performing a forward Domain Name System (DNS) lookup to gather a domain name for the IP address.
9. The method of claim 1 , wherein the identifier comprises a hostname or domain name, and the method further comprises after the parsing step:
performing a reverse DNS lookup to gather an IP address.
10. A computer-implemented method for use with log messages generated by one or more platforms of a data system, comprising the steps of:
receiving; at a log manager, from an agent for acquiring log messages associated with the one or more platforms, a request for classification information for a log message, wherein the classification information represents a type of device or process responsible for generating the log message, and wherein the request includes an identifier obtained from the log message;
determining, at the log manager, whether the identifier can be mapped to classification information; and
sending classification information to the agent responsive to the log manager determining that the identifier can be mapped to classification information, wherein the determination that the identifier can be mapped to classification information comprises at least one of a) identifying classification information in a database that corresponds to the identifier and b) creating new classification information and assigning the identifier to the newly created classification information.
11. The method of claim 10 , further comprising:
returning a failed classification information request message to the log agent responsive to the log manager determining that the identifier cannot be mapped to classification information.