Method and apparatus for retrieving and combining summarized log data in a distributed log data processing system
View Patent ↗A system can receive raw log data from log-producing devices. The system can parse the raw log data to generate parsed log data, including extracting fields from the raw log data, generating structured query language (SQL) statements from the extracted fields, and inserting the SQL statements in a database. The system can summarize the parsed log data, including identifying one or more SQL statements based on fields of commonality and fields of uniqueness, and creating summarized data that include a new statement condensed from the identified SQL statements. The system can then generate a report based on the summarized data.
1. A method for processing log data, the method comprising:
receiving raw log data in a log data analyzer from a plurality of log-producing devices, the raw log data including a plurality of messages;
parsing the raw log data to generate structured query language (SQL) statements representing the raw log data, including:
extracting fields from the raw log data based on a log data message type for each message, the log data message type indicating that a respective message is associated with a pre-defined type of event on a log-producing device that produced the respective message;
generating the SQL statements from the extracted fields, wherein the extracted fields are incorporated into the SQL statements; and
inserting the SQL statements into a database table designated to store the generated SQL statements on a volatile storage device, the database table being different from database tables for storing the raw log data;
summarizing the SQL statements, including:
at pre-specified time intervals, copying the SQL statements stored in the database table stored on the volatile storage device to one or more database tables stored on a non-volatile storage device;
identifying, from the SQL statements stored in the one or more database tables on the non-volatile storage device, a set of one or more SQL statements each including one or more fields of commonality and one or more fields of uniqueness, each field of commonality being a field storing a value that is commonly incorporated in the set of SQL statements, each field of uniqueness being a field storing values that are different among the set of SQL statements; and
creating summarized data, the summarized data including a new statement condensed from the identified set of one or more SQL statements based on the one or more fields of commonality and one or more fields of uniqueness; and
generating a report based on the summarized data, wherein the method is performed by one or more computers.
2. The method of claim 1 , where the one or more fields of commonality include at least one of an identical source Internet protocol (IP) address, an identical destination IP address, or an identical destination port number.
3. The method of claim 1 , where the one or more fields of uniqueness includes a source port number.
4. The method as recited in claim 1 , where inserting the SQL statements into the database table includes inserting the SQL statements into an accept table or a deny table.
5. The method of claim 4 , where each of the accept table and deny table is configured to store data over a time period.
6. The method of claim 1 , where the data in the report are limited by a time period of the summarized data.
7. The method of claim 1 , wherein generating the report based on the summarized data comprises:
receiving a database query from a management station;
generating a database report in the log data analyzer using the summarized data in response to the database query; and
sending the database report to the management station.
8. A data processing system for processing log data comprising:
a management station including one or more processors; and
a log data analyzer including one or more processors, the log data analyzer being connected to the management station via a data communications link and configured to perform operations comprising:
receiving raw log data from a plurality of log-producing devices, the raw log data including a plurality of messages;
parsing the raw log data to generate structured query language (SQL) statements representing the raw log data, including:
extracting fields from the raw log data based on a log data message type for each message, the log data message type indicating that a respective message is associated with a pre-defined type of event on a log-producing device that produced the respective message;
generating the SQL statements from the extracted fields, wherein the extracted fields are incorporated into the SQL statements; and
inserting the SQL statements into a database table designated to store the generated SQL statements, the database table being different from database tables for storing the raw log data;
summarizing the SQL statements, including:
identifying, from the SQL statements inserted into the database table, a set of one or more SQL statements based on one or more fields of commonality and one or more fields of uniqueness, each field of commonality being a field storing a value that is commonly incorporated in the set of SQL statements, each field of uniqueness being a field storing values that are different among the set of SQL statements; and
creating summarized data, the summarized data including a new statement condensed from the set of one or more SQL statement based on the one or more fields of commonality and one or more fields of uniqueness; and
generating a report based on the summarized data.
9. The system of claim 8 , where the one or more fields of commonality include at least one of an identical source Internet protocol (IP) address, an identical destination IF address, or an identical destination port number.
10. The system of claim 8 , where the one or more fields of uniqueness includes a source port number.
11. The system of claim 8 , where inserting the SQL statements into the storing includes inserting the SQL statements into an accept table or a deny table.
12. The system of claim 11 , where each of the accept table and deny table is configured to store data over a time period.
13. The system of claim 8 wherein the log data analyzer limits the data in the report by a time period of the summarized data.
14. The system of claim 8 , wherein generating the report based on the summarized data comprises:
receiving a database query from the management station;
generating a database report in the log data analyzer using the summarized data in response to the database query; and
sending the database report to the management station.
15. A non-transitory storage device storing computer software, the computer software operable to cause one or more computers to perform operations comprising:
receiving raw log data in a log data analyzer from a plurality of log-producing devices, the raw log data including a plurality of messages;
parsing the raw log data to generate structured query language (SQL) statements representing the raw log data, including:
extracting fields from the raw log data based on a log data message type for each message, the log data message type indicating that a respective message is associated with a pre-defined type of event on a log-producing device that produced the respective message;
generating the SQL statements from the extracted fields, wherein the extracted fields are incorporated into the SQL statements; and
inserting the SQL statements into a database table designated to store the generated SQL statements the database table being different from database tables for storing the raw log data;
summarizing the SQL statements, including:
identifying, from the SQL statements inserted into the database table, a set of one or more SQL statements based on one or more fields of commonality and one or more fields of uniqueness, each field of commonality being a field storing a value that is commonly incorporated in the set of SQL statements, each field of uniqueness being a field storing values that are different among the set of SQL statements; and
creating summarized data, the summarized data including a new statement condensed from the identified set of one or more SQL statements based on the one or more fields of commonality and one or more fields of uniqueness; and
generating a report based on the summarized data.
16. The non-transitory storage device of claim 15 , where the one or more fields of commonality include at least one of an identical source Internet protocol (IP) address, an identical destination IP address, or an identical destination port number.
17. The non-transitory storage device of claim 15 , where the one or more fields of uniqueness includes a source port number.
18. The non-transitory storage device as recited in claim 15 , where inserting the SQL statements into the database table includes inserting the SQL statements into an accept table or a deny table.
19. The non-transitory storage device of claim 18 , where each of the accept table and deny table is configured to store data over a time period.