System and method for parsing, summarizing and reporting log data
View Patent ↗A system and method is disclosed which enables network administrators and the like to quickly analyze the data produced by log-producing devices such as network firewalls and routers. Unlike systems of the prior art, the system disclosed herein automatically parses and summarizes log data before inserting it into one or more databases. This greatly reduces the volume of data stored in the database and permits database queries to be run and reports generated while many types of attempted breaches of network security are still in progress. Database maintenance may also be accomplished automatically by the system to delete or archive old log data.
1. A data processing system comprising:
one or more computers;
a log-producing device connected to the one or more computers through a network; and,
a non-transitory storage device storing computer instructions operable to cause the one or more computers to perform operations comprising:
receiving, by a log receiver, a log message from the log-producing device;
determining, by the log receiver, a property of the log message;
copying, by the log receiver, the log message into a first storage buffer;
parsing, by a log parser, content of the first storage buffer, including:
extracting data fields from the log message in the first storage buffer; and
converting the extracted data fields into one or more structured query language statements;
copying, by the log parser, the one or more structured query language statements into a second storage buffer;
reading, by a database inserter, the one or more structured query language statements in the second storage buffer, wherein reading the one or more structured query language statements comprises examining each of the one or more structured query language statements by the database inserter to determine a respective database table for storing a corresponding structured query language statement of the one of more structured query language statements;
inserting, by the database inserter, each of the one or more structured query language statements into a corresponding database table; and
summarizing, by a message collection engine, the one or more structured query language statements in the one or more database tables into summarized statements for storing on the non-transitory storage device.
2. The data processing system of claim 1 , wherein determining the property of the log message comprises:
determining whether the log-producing device is an approved data source, wherein the determining comprises comparing the log-producing device with a group of acceptable data sources;
determining whether the log-producing device is enabled and configured; and
upon the determining that the log-producing device is an approved data source and that the log-producing device is enabled and configured, approving the log message for parsing.
3. The data processing system of claim 2 , wherein determining the property of the log message comprises:
upon the determining that the log-producing device is not an approved data source, or upon the determining that the log-producing device not configured, copying, by the log receiver, the log message into an unapproved ring buffer configured to store a given number of log messages.
4. The data processing system of claim 1 , wherein:
each of the first storage buffer and the second storage buffer is a first-in-first-out ring buffer; and
at least one of the first storage buffer or the second storage buffer includes a real-time view buffer configured to store the log message for real-time viewing.
5. The data processing system of claim 1 , wherein extracting the data fields from the log message in the first storage buffer comprises:
searching the log message for one or more predetermined keywords;
determining a message type of the log message based on a result of the searching; and
extracting the data fields using a pre-determined function associated with the message type.
6. The data processing system of claim 1 , wherein:
each database table comprises at least one of an accept table, a deny table, a security table, a system table, a uniform resource locator (URL) table, or a file transfer protocol (FTP) table, and
inserting each of the one or more structured query language statements into the corresponding database table comprises:
inserting the structured query language statement into the deny table when the log message corresponding to the structured query language statement indicates a denial by the log-producing device;
inserting the structured query language statement into the system table when the log message corresponding to the structured query language statement indicates a system activity of the log-producing device;
inserting the structured query language statement into the URL table when the log message corresponding to the structured query language statement relates to a network user accessing a URL site;
inserting the structured query language statement into the FTP table when the log message corresponding to the structured query language statement relates to a network user requesting FTP service, or
inserting the structured query language statement into the accept table.
7. The data processing system of claim 1 , wherein the second storage buffer is configured to receive structured query language statements from a plurality of procedures and to store the received structured query language statements in a queue.
8. A method comprising:
receiving, by a log receiver, a log message from the log-producing device;
determining, by the log receiver, a property of the log message;
copying, by the log receiver, the log message into a first storage buffer;
parsing, by a log parser, content of the first storage buffer, including:
extracting data fields from the log message in the first storage buffer; and
converting the extracted data fields into one or more structured query language statements;
copying, by the log parser, the one or more structured query language statements into a second storage buffer;
reading, by a database inserter, the one or more structured query language statements in the second storage buffer, wherein reading the one or more structured query language statements comprises examining each of the one or more structured query language statements by the database inserter to determine a respective database table for storing a corresponding structured query language statement of the one or more structured query language statements;
inserting, by the database inserter, each of the one or more structured query language statements into a corresponding database table; and
summarizing, by a message collection engine, the one or more structured query language statements in the one or more database tables into summarized statements for storing on a non-transitory storage device,
wherein the method is performed by one or more computers.
9. The method of claim 8 , wherein determining the property of the log message comprises:
determining whether the log-producing device is an approved data source, wherein the determining comprises comparing the log-producing device with a group of acceptable data sources;
determining whether the log-producing device is enabled and configured; and
upon the determining that the log-producing device is an approved data source and that the log-producing device is enabled and configured, approving the log message for parsing.
10. The method of claim 9 , wherein determining the property of the log message comprises:
upon the determining that the log-producing device is not an approved data source, or upon the determining that the log-producing device not configured, copying, by the log receiver, the log message into an unapproved ring buffer configured to store a given number of log messages.
11. The method of claim 8 , wherein:
each of the first storage buffer and the second storage buffer is a first-in-first-out ring buffer; and
at least one of the first storage buffer or the second storage buffer includes a real-time view buffer configured to store the log message for real-time viewing.
12. The method of claim 8 , wherein extracting the data fields from the log message in the first storage buffer comprises:
searching the log message for one or more predetermined keywords;
determining a message type of the log message based on a result of the searching; and
extracting the data fields using a pre-determined function associated with the message type.
13. The method of claim 8 , wherein:
each database table comprises at least one of an accept table, a deny table, a security table, a system table, a uniform resource locator (URL) table, or a file transfer protocol (FTP) table, and
inserting each of the one or more structured query language statements into the corresponding database table comprises:
inserting the structured query language statement into the deny table when the log message corresponding to the structured query language statement indicates a denial by the log-producing device;
inserting the structured query language statement into the system table when the log message corresponding to the structured query language statement indicates a system activity of the log-producing device;
inserting the structured query language statement into the URL table when the log message corresponding to the structured query language statement relates to a network user accessing a URL site;
inserting the structured query language statement into the FTP table when the log message corresponding to the structured query language statement relates to a network user requesting FTP service, or
inserting the structured query language statement into the accept table.
14. The method of claim 8 , wherein the second storage buffer is configured to receive structured query language statements from a plurality of procedures and to store the received structured query language statements in a queue.
15. A non-transitory storage device storing computer instructions operable to cause one or more computers to perform operations comprising:
receiving, by a log receiver, a log message from the log-producing device;
determining, by the log receiver, a property of the log message;
copying, by the log receiver, the log message into a first storage buffer;
parsing, by a log parser, content of the first storage buffer, including:
extracting data fields from the log message in the first storage buffer; and
converting the extracted data fields into one or more structured query language statements;
copying, by the log parser, the one or more structured query language statements into a second storage buffer;
reading, by a database inserter, the one or more structured query language statements in the second storage buffer, wherein reading the one or more structured query language statements comprises examining each of the one or more structured query language statements by the database inserter to determine a respective database table for storing a corresponding structured query language statement of the one or more structured query language statements;
inserting, by the database inserter, each of the one or more structured query language statements into a corresponding database table; and
summarizing, by a message collection engine, the one or more structured query language statements in the one or more database tables into summarized statements for storing on the non-transitory storage device.
16. The non-transitory storage device of claim 15 , wherein determining the property of the log message comprises:
determining whether the log-producing device is an approved data source, wherein the determining comprises comparing the log-producing device with a group of acceptable data sources;
determining whether the log-producing device is enabled and configured; and
upon the determining that the log-producing device is an approved data source and that the log-producing device is enabled and configured, approving the log message for parsing.
17. The non-transitory storage device of claim 16 , wherein determining the property of the log message comprises:
upon the determining that the log-producing device is not an approved data source, or upon the determining that the log-producing device not configured, copying, by the log receiver, the log message into an unapproved ring buffer configured to store a given number of log messages.
18. The non-transitory storage device of claim 15 , wherein:
each of the first storage buffer and the second storage buffer is a first-in-first-out ring buffer; and
at least one of the first storage buffer or the second storage buffer includes a real-time view buffer configured to store the log message for real-time viewing.
19. The non-transitory storage device of claim 15 , wherein extracting the data fields from the log message in the first storage buffer comprises:
searching the log message for one or more predetermined keywords;
determining a message type of the log message based on a result of the searching; and
extracting the data fields using a pre-determined function associated with the message type.
20. The non-transitory storage device of claim 15 , wherein:
each database table comprises at least one of an accept table, a deny table, a security table, a system table, a uniform resource locator (URL) table, or a file transfer protocol (FTP) table, and
inserting each of the one or more structured query language statements into the corresponding database table comprises:
inserting the structured query language statement into the deny table when the log message corresponding to the structured query language statement indicates a denial by the log-producing device;
inserting the structured query language statement into the system table when the log message corresponding to the structured query language statement indicates a system activity of the log-producing device;
inserting the structured query language statement into the URL table when the log message corresponding to the structured query language statement relates to a network user accessing a URL site;
inserting the structured query language statement into the FTP table when the log message corresponding to the structured query language statement relates to a network user requesting FTP service, or
inserting the structured query language statement into the accept table.
21. The non-transitory storage device of claim 15 , wherein the second storage buffer is configured to receive structured query language statements from a plurality of procedures and to store the received structured query language statements in a queue.