Querying distributed log data using virtual fields defined in query strings
The present disclosure relates to querying log messages stored at log endpoints (e.g., log cores) using query strings that reference virtual fields, and generating log field views presenting aggregate information for groups of log messages that are grouped based on criteria defined in the query string.
1. A computer-implemented method, comprising:
receiving a plurality of log messages, wherein each log message of the plurality of log messages includes message data, and wherein the message data includes one or more detectable signatures to segment the message data into data fields;
mapping each log message of the plurality of log messages to a message structure, wherein the message structure defines a fixed portion or a variable portion of the message data as a data field;
detecting, for each log message of the plurality of log messages, one or more data fields within the message data, based at least in part on the one or more detectable signatures included in the message data and the mapping of the log message to the message structure, wherein each data field of the one or more data fields corresponds to a value present in the message data;
receiving, via a user interface, input corresponding to a query string for querying at least a portion of the plurality of log messages, the query string including one or more virtual data fields, wherein each of the one or more virtual data fields corresponds to a function of at least one data field of the detected one or more data fields, and wherein the one or more virtual data fields are not present in the detected one or more data fields or the message structure;
parsing the query string to identify each of the one or more virtual data fields;
in response to the parsing, formulating a response to the query string at least in part by:
querying one or more log data stores using the at least one data field, wherein the plurality of log messages are stored at the one or more log data stores,
receiving, for each of the one or more log data stores, partial result data in response to the querying of the log data store, the partial result data including one or more results of querying log messages stored at the log data store,
aggregating the partial result data received from each of the one or more log data stores to determine a preliminary result for the at least one data field, and
performing the function on the preliminary result to determine a formulated response to the query string that included the one or more virtual data fields;
generating a log field view representing the formulated response to the query string, wherein the formulated response to the query string presents the preliminary result; and
presenting, at the user interface, the log field view.
2. The computer-implemented method of claim 1 , wherein a virtual data field of the one or more virtual data fields is based on a start of a transaction or an end of the transaction, wherein the transaction corresponds to a transaction identifier, wherein the transaction identifier is a detectable field in a log message of the plurality of log messages, and wherein the start and/or the end of the transaction is computed based on a set of log messages associated with the transaction.
3. The computer-implemented method of claim 1 , wherein presenting the log field view at the interface includes presenting a table including each of the one or more virtual data fields, such that each of the one or more virtual data fields includes an associated one or more virtual data elements, and each of the one or more virtual data elements includes a value.
4. The computer-implemented method of claim 3 , further comprising:
receiving, at the interface, additional input corresponding to a selection of a virtual data element from amongst the one or more virtual data fields presented in the table; and
identifying at least one log message associated with the selected virtual data element; and
presenting information associated with each of the at least one identified log message.
5. The computer-implemented method of claim 1 , further comprising:
performing a group-by operation on the aggregated partial result data using the one or more virtual fields, wherein performing the group-by operation includes grouping log messages of the plurality of log messages into one or more groups, wherein log messages included in a group share a same attribute with other log messages in the group.
6. The computer-implemented method of claim 1 , wherein the plurality of log messages includes each of a first log message and a second log message, wherein the first log message is in a first format and the second log message is in a second format, and wherein the first format is different from the second format.
7. The computer-implemented method of claim 6 , wherein the first log message includes a particular data field at a first position within the first log message, wherein the second log message includes the particular data field at a second position within the second log message, and wherein the first position is different from the second position.
8. The computer-implemented method of claim 6 , wherein the first log message includes a particular data field represented using a first field name within the first log message, wherein the second log message includes the particular data field represented using a second field name within the second log message, and wherein the first field name is different from the second field name.
9. A system, comprising:
one or more data processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations including:
receiving a plurality of log messages, wherein each log message of the plurality of log messages includes message data, and wherein the message data includes one or more detectable signatures to segment the message data into data fields;
mapping each log message of the plurality of log messages to a message structure, wherein the message structure defines a fixed portion or a variable portion of the message data as a data field;
detecting, for each log message of the plurality of log messages, one or more data fields within the message data, based at least in part on the one or more detectable signatures included in the message data and the mapping of the log message to the message structure, wherein each data field of the one or more data fields corresponds to a value present in the message data;
receiving, via a user interface, input corresponding to a query string for querying at least a portion of the plurality of log messages, the query string including one or more virtual data fields, wherein each of the one or more virtual data fields corresponds to a function of at least one data field of the detected one or more data fields, and wherein the one or more virtual data fields are not present in the detected one or more data fields or the message structure,
parsing the query string to identify each of the one or more virtual data fields;
in response to the parsing, formulating a response to the query string at least in part by:
querying one or more log data stores using the at least one data field, wherein the plurality of log messages are stored at the one or more log data stores,
receiving, for each of the one or more log data stores, partial result data in response to the querying of the log data store, the partial result data including one or more results of querying log messages stored at the log data store,
aggregating the partial result data received from each of the one or more log data stores to determine a preliminary result for the at least one data field, and
performing the function on the preliminary result to determine a formulated response to the query string that included the one or more virtual data fields;
generating a log field view representing the formulated response to the query string, wherein the formulated response to the query string presents the preliminary result; and
presenting, at the user interface, the log field view.
10. The system of claim 9 , wherein a virtual data field of the one or more virtual data fields is based on a start of a transaction or an end of the transaction, wherein the transaction corresponds to a transaction identifier, wherein the transaction identifier is a detectable field in a log message of the plurality of log messages, and wherein the start and/or the end of the transaction is computed based on a set of log messages associated with the transaction.
11. The system of claim 9 , wherein presenting the log field view at the interface includes presenting a table including each of the one or more virtual data fields, such that each of the one or more virtual data fields includes an associated one or more virtual data elements, and each of the one or more virtual data elements includes a value.
12. The system of claim 11 , further comprising:
receiving, at the interface, additional input corresponding to a selection of a virtual data element from amongst the one or more virtual data fields presented in the table; and
identifying at least one log message associated with the selected virtual data element; and presenting information associated with each of the at least one identified log message.
13. The system of claim 9 , wherein the operations further comprise:
performing a group-by operation on the aggregated partial result data using the one or more virtual fields, wherein performing the group-by operation includes grouping log messages of the plurality of log messages into one or more groups, wherein log messages included in a group share a same attribute with other log messages in the group.
14. The system of claim 9 , wherein the plurality of log messages includes each of a first log message and a second log message, wherein the first log message is in a first format and the second log message is in a second format, and wherein the first format is different from the second format.
15. The system of claim 14 , wherein the first log message includes a particular data field at a first position within the first log message, wherein the second log message includes the particular data field at a second position within the second log message, and wherein the first position is different from the second position.
16. The system of claim 14 , wherein the first log message includes a particular data field represented using a first field name within the first log message, wherein the second log message includes the particular data field represented using a second field name within the second log message, and wherein the first field name is different from the second field name.
17. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations including:
receiving a plurality of log messages, wherein each log message of the plurality of log messages includes message data, and wherein the message data includes one or more detectable signatures to segment the message data into data fields;
mapping each log message of the plurality of log messages to a message structure, wherein the message structure defines a fixed portion or a variable portion of the message data as a data field;
detecting, for each log message of the plurality of log messages, one or more data fields within the message data, based at least in part on the one or more detectable signatures included in the message data and the mapping of the log message to the message structure, wherein each data field of the one or more data fields corresponds to a value present in the message data;
receiving, via a user interface, input corresponding to a query string for querying at least a portion of the plurality of log messages, the query string including one or more virtual data fields, wherein each of the one or more virtual data fields corresponds to a function of at least one data field of the detected one or more data fields, and wherein the one or more virtual data fields are not present in the detected one or more data fields or the message structure;
parsing the query string to identify each of the one or more virtual data fields;
in response to the parsing, formulating a response to the query string at least in part by:
querying one or more log data stores using the at least one data field, wherein the plurality of log messages are stored at the one or more log data stores,
receiving, for each of the one or more log data stores, partial result data in response to the querying of the log data store, the partial result data including one or more results of querying log messages stored at the log data store,
aggregating the partial result data received from each of the one or more log data stores to determine a preliminary result for the at least one data field, and
performing the function on the preliminary result to determine a formulated response to the query string that included the one or more virtual data fields;
generating a log field view representing the formulated response to the query string, wherein the formulated response to the query string presents the preliminary result; and
presenting, at the user interface, the log field view.
18. The computer-program product of claim 17 , wherein a virtual data field of the one or more virtual data fields is based on a start of a transaction or an end of the transaction, wherein the transaction corresponds to a transaction identifier, wherein the transaction identifier is a detectable field in a log message of the plurality of log messages, and wherein the start and/or the end of the transaction is computed based on a set of log messages associated with the transaction.
19. The computer-program product of claim 17 , wherein presenting the log field view at the interface includes presenting a table including each of the one or more virtual data fields, such that each of the one or more virtual data fields includes an associated one or more virtual data elements, and each of the one or more virtual data elements includes a value.
20. The computer-program product of claim 19 , further comprising:
receiving, at the interface, additional input corresponding to a selection of a virtual data element from amongst the one or more virtual data fields presented in the table; and
identifying at least one log message associated with the selected virtual data element; and
presenting information associated with each of the at least one identified log message.
21. The computer-program product of claim 17 , further comprising:
performing a group-by operation on the aggregated partial result data using the one or more virtual fields, wherein performing the group-by operation includes grouping log messages of the plurality of log messages into one or more groups, wherein log messages included in a group share a same attribute with other log messages in the group.
22. The computer-program product of claim 17 , wherein the plurality of log messages includes each of a first log message and a second log message, wherein the first log message is in a first format and the second log message is in a second format, and wherein the first format is different from the second format.
23. The computer-program product of claim 22 , wherein the first log message includes a particular data field at a first position within the first log message, wherein the second log message includes the particular data field at a second position within the second log message, and wherein the first position is different from the second position.