Methods and systems for storing and querying log messages using log message bifurcation
Methods and systems described herein are directed to storing and querying log messages using log message bifurcation. Each log message that exceeds a time limit for storage in a live storage database is bifurcated into a log pattern that is stored in a log-pattern database and a metric object that records variable information of the log message in a time-series metric database. The log-pattern database and the time-series metric database together provide long-term log storage for the information contained in log messages that are too old to be stored in a live storage database. A request for access to log messages stored in the long-term log storage is completed by reconstructing the requested log messages from log patterns stored in the log-pattern database and metric objects stored in the time-series metric database.
1. A method stored in one or more data-storage devices and executed using one or more processors of a computer system for storing and querying log messages generated by event sources in a distributed computing system, the method comprising:
storing log messages generated by the event sources in a live storage database;
for each log message of the live storage database that exceeds a time limit for storage in the live storage database and has a corresponding log pattern in a log-pattern database, getting a log identification (ID) for the log message from a log-pattern table stored in the log-pattern database;
for each log message of the live storage database that exceeds the time limit and does not have a corresponding log pattern in the log-pattern database,
determining fixed segments and variable segments of the log message,
determining a log ID corresponding to a new class of log messages that includes the log message,
determining a log pattern from the fixed segments and placeholders substituted for the variable segments of the log message, and
storing the log pattern in the log-pattern database;
for each log message of the live storage database that exceeds the time limit, constructing a metric object from variable segments of the log message and the log ID of the log message;
storing the metric object in a time-series metric database;
retrieving log patterns from the log-pattern database and metric objects from the time-series metric database in response to a request for log messages in a query time interval; and
reconstructing log messages with time stamps in the query time interval from the log patterns retrieved from the log-pattern database and the metric objects retrieved from the time-series metric database.
2. The method of claim 1 wherein retrieving log patterns from the log-pattern database and metric objects from the time-series metric database in response to a request for log messages in a query time interval: receiving a query time interval via a user interface; identifying metric objects in the time-series database with corresponding time stamps in the query time interval; reading log IDs recorded in each metric object of the metric objects; and retrieving a log pattern from the log-pattern database for each different log ID.
3. The method of claim 1 wherein reconstructing log messages with time stamps in the query time interval comprises: for each log pattern with a log ID that corresponds to at least one metric object with a time stamp in the query time interval, for each metric object of the metric objects with the log ID, replacing placeholders in the log pattern with corresponding variable segments of the metric object to obtain a log message with a time stamp in the query time interval.
4. The method of claim 1 wherein reconstructing log messages with the time stamps in the query time interval comprises the query time interval not overlapping a recent time interval for retaining log messages in the live storage database.
5. The method of claim 1 wherein reconstructing log messages with time stamps in the query time interval comprises: reconstructing log messages with time stamps in a portion of the query time interval that does not overlap a recent time interval for retaining log messages in the live storage database; and copying log messages with time stamps in a portion of the query time interval that overlaps with the recent time interval from the live storage database.
6. The method of claim 1 further comprises displaying reconstructed log messages with time stamps in the query time interval in graphical user interface.
7. A computer system for storing and querying log messages generated by event sources in a distributed computing system, the system comprising:
one or more processors;
one or more data-storage devices; and
machine-readable instructions stored in the one or more data-storage devices that when executed using the one or more processors controls the computer system to perform operations comprising:
storing log messages generated by the event sources in a live storage database;
for each log message of the live storage database that exceeds a time limit for storage in the live storage database and has a corresponding log pattern in a log-pattern database, getting a log identification (ID) for the log message from a log-pattern table stored in the log-pattern database;
for each log message of the live storage database that exceeds the time limit and does not have a corresponding log pattern in the log-pattern database,
determining fixed segments and variable segments of the log message,
determining a log ID corresponding to a new class of log messages that includes the log message,
determining a log pattern from the fixed segments and placeholders substituted for the variable segments of the log message, and
storing the log pattern in the log-pattern database;
for each log message of the live storage database that exceeds the time limit, constructing a metric object from variable segments of the log message and the log ID of the log message;
storing the metric object in a time-series metric database;
retrieving log patterns from the log-pattern database and metric objects from the time-series metric database in response to a request for log messages in a query time interval; and
reconstructing log messages with time stamps in the query time interval from the log patterns retrieved from the log-pattern database and the metric objects retrieved from the time-series metric database.
8. The computer system of claim 7 wherein retrieving log patterns from the log-pattern database and metric objects from the time-series metric database in response to a request for log messages in a query time interval: receiving a query time interval via a user interface; identifying metric objects in the time-series database with corresponding time stamps in the query time interval; reading log IDs recorded in each metric object of the metric objects; and retrieving a log pattern from the log-pattern database for each different log ID.
9. The computer system of claim 7 wherein reconstructing log messages with time stamps in the query time interval comprises: for each log pattern with a log ID that corresponds to at least one metric object with a time stamp in the query time interval, for each metric object of the metric objects with the log ID, replacing placeholders in the log pattern with corresponding variable segments of the metric object to obtain a log message with a time stamp in the query time interval.
10. The computer system of claim 1 wherein reconstructing log messages with the time stamps in the query time interval comprises the query time interval not overlapping a recent time interval for retaining log messages in the live storage database.
11. The computer system of claim 7 wherein reconstructing log messages with time stamps in the query time interval comprises: reconstructing log messages with time stamps in a portion of the query time interval that does not overlap a recent time interval for retaining log messages in the live storage database; and
copying log messages with time stamps in a portion of the query time interval that overlaps with the recent time interval from the live storage database.
12. The computer system of claim 7 further comprises displaying reconstructed log messages with time stamps in the query time interval in a graphical user interface.
13. A non-transitory computer-readable medium encoded with machine-readable instructions that implement a method carried out by one or more processors of a computer system to perform operations comprising:
storing log messages generated by event sources in a live storage database;
for each log message of the live storage database that exceeds a time limit for storage in the live storage database and has a corresponding log pattern in a log-pattern database, getting a log identification LID, for the log message from a log-pattern table stored in the log-pattern database;
for each log message of the live storage database that exceeds the time limit and does not have a corresponding log pattern in the log-pattern database,
determining fixed segments and variable segments of the log message,
determining a log ID corresponding to a new class of log messages that includes the log message,
determining a log pattern from the fixed segments and placeholders substituted for the variable segments of the log message, and
storing the log pattern in the log-pattern database;
for each log message of the live storage database that exceeds the time limit, constructing a metric object from variable segments of the log message and the log ID of the log message;
storing the metric object in a time-series metric database;
retrieving log patterns from the log-pattern database and metric objects from the time-series metric database in response to a request for log messages in a query time interval; and
reconstructing log messages with time stamps in the query time interval from the log patterns retrieved from the log-pattern database and the metric objects retrieved from the time-series metric database.
14. The medium of claim 13 wherein retrieving log patterns from the log-pattern database and metric objects from the time-series metric database in response to a request for log messages in a query time interval: receiving a query time interval via a user interface; identifying metric objects in the time-series database with corresponding time stamps in the query time interval; reading log IDs recorded in each metric object of the metric objects; and retrieving a log pattern from the log-pattern database for each different log ID.
15. The medium of claim 13 wherein reconstructing log messages with time stamps in the query time interval comprises: for each log pattern with a log ID that corresponds to at least one metric object with a time stamp in the query time interval, for each metric object of the metric objects with the log ID, replacing placeholders in the log pattern with corresponding variable segments of the metric object to obtain a log message with a time stamp in the query time interval.
16. The medium of claim 13 wherein reconstructing log messages with the time stamps in the query time interval comprises the query time interval not overlapping a recent time interval for retaining log messages in the live storage database.
17. The medium of claim 13 wherein reconstructing log messages with time stamps in the query time interval comprises: reconstructing log messages with time stamps in a portion of the query time interval that does not overlap a recent time interval for retaining log messages in the live storage database; and copying log messages with time stamps in a portion of the query time interval that overlaps with the recent time interval from the live storage database.
18. The medium of claim 13 further comprises displaying reconstructed log messages with time stamps in the query time interval in a graphical user interface.