Clustering of structured log data by key-values
Clustering structured log data by key-values includes receiving, via a user interface, a request to apply an operator to cluster a set of raw log messages according to values for a set of keys associated with the request. At least a portion of each raw log message comprises structured machine data including a set of key-value pairs. It further includes receiving a raw log message in the set of raw log messages. It further includes determining whether to include the raw log message in a cluster based at least in part on an evaluation of values in the structured machine data of the raw log message for the set of keys associated with the request. The cluster is included in a plurality of clusters. Each cluster in the plurality is associated with a different combination of values for the set of keys associated with the request. It further includes providing, via the user interface, information associated with the cluster.
1. A system, comprising:
a processor configured to:
receive, via a user interface (UI), a first request to cluster a set of log messages based on a set of keys included in the first request, wherein each log message comprises machine data in a set of key-value pairs, each key defining a structural parameter in the log message, the key-value pair comprising an identifier of the key and the value for the key in the log message;
cluster, in response to the first request, the log messages into a first set of clusters, each cluster from the first set of clusters comprising log messages that include a unique combination of keys, the log messages in each cluster comprising key-value pairs with an identifier of one key from the unique combination of keys of the cluster and any value;
present, in the UI, information of the first set of clusters, the information comprising the unique combination of keys for each cluster, the UI providing an option to further cluster one of the first set of clusters based on the values in the key-value pairs associated with the cluster;
receive, via the UI, a second request for information about a given cluster from the first set of clusters;
cluster the log messages from the given cluster into a second set of clusters based on the values in the key-value pairs in the log messages of the given cluster, wherein each cluster in the second set of clusters is associated with a different combination of values for the unique combination of keys associated with the given cluster; and
provide, via the IA, information associated with the second set of clusters; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system recited in claim 1 wherein the presented information associated with the first set of clusters comprises a count of log messages included in each cluster from the first set of clusters.
3. The system recited in claim 1 wherein the presented information of the second set of clusters comprises a number of log messages included in each cluster from the second set of clusters.
4. The system recited in claim 1 wherein presenting information of the first set of clusters further comprises to:
present, via the UI, a link to view log messages included in each cluster from the first set of clusters.
5. The system recited in claim 1 wherein the processor is further configured to:
generate a signature for the given cluster and present the signature in the UI, wherein the signature is generated based at least in part on a set of values for the set of keys associated with the given cluster.
6. A method, comprising:
receiving, via a user interface (UI), a first request to cluster a set of log messages based on a set of keys included in the first request, wherein each log message comprises machine data in a set of key-value pairs, each key defining a structural parameter in the log message, the key-value pair comprising an identifier of the key and the value for the key in the log message;
clustering, in response to the first request, the log messages into a first set of clusters, each cluster from the first set of clusters comprising log messages that include a unique combination of keys, the log messages in each cluster comprising key-value pairs with an identifier of one key from the unique combination of keys of the cluster and any value;
presenting, in the UI, information of the first set of clusters, the information comprising the unique combination of keys for each cluster, the UI providing an option to further cluster one of the first set of clusters based on the values in the key-value pairs associated with the cluster;
receiving, via the UI, a second request for information about a given cluster from the first set of clusters;
clustering the log messages from the given cluster into a second set of clusters based on the values in the key-value pairs in the log messages of the given cluster, wherein each cluster in the second set of clusters is associated with a different combination of values for the unique combination of keys associated with the given cluster; and
providing, via the UI, information associated with the second set of clusters.
7. The method of claim 6 wherein the presented information associated with the first set of clusters comprises a count of log messages included in each cluster from the first set of clusters.
8. The method of claim 6 wherein the presented information of the second set of dusters comprises a number of log messages included in each cluster from the second set of clusters.
9. The method of claim 6 wherein presenting information of the first set of clusters further comprises:
presenting, via the UI, a link to view log messages included in each cluster from the first set of clusters.
10. The method of claim 6 further comprising:
generating a signature for the given duster and present the signature in the UI, wherein the signature is generated based at least in part on a set of values for the set of keys associated with the given cluster.
11. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving, via a user interface (UI), a first request to cluster a set of log messages based on a set of keys included in the first request, wherein each log message comprises machine data in a set of key-value pairs, each key defining a structural parameter in the log message, the key-value pair comprising an identifier of the key and the value for the key in the log message;
clustering, in response to the first request, the log messages into a first set of clusters, each cluster from the first set of clusters comprising log messages that include a unique combination of keys, the log messages in each cluster comprising key-value pairs with an identifier of one key from the unique combination of keys of the cluster and any value;
presenting, in the UI, information of the first set of clusters, the information comprising the unique combination of keys for each cluster, the UI providing an option to further cluster one of the first set of clusters based on the values in the key-value pairs associated with the cluster;
receiving, via the UI, a second request for information about a given cluster from the first set of dusters;
cluster the log messages from the given cluster into a second set of clusters based on the values in the key-value pairs in the log messages of the given cluster, wherein each cluster in the second set of clusters is associated with a different combination of values for the unique combination of keys associated with the given cluster; and
providing, via the UI, information associated with the second set of clusters.
12. The computer program product embodied in a non-transitory computer readable storage medium as recited in claim 11 , wherein the presented information of the second set of clusters comprises a number of log messages included in each cluster from the second set of clusters.
13. The computer program product embodied in a non-transitory computer readable storage medium as recited in claim 11 , wherein the presented information of the second set of clusters comprises a number of log messages included in each cluster from the second set of clusters.
14. The computer program product embodied in a non-transitory computer readable storage medium as recited in claim 11 , wherein presenting information of the first set of clusters further comprises:
presenting, via the UI, a link to view log messages included in each cluster from the first set of clusters.
15. The computer program product embodied in a non-transitory computer readable storage medium as recited in claim 11 further comprising instructions for:
generating a signature for the given cluster and present the signature in the UI, wherein the signature is generated based at least in part on a set of values for the set of keys associated with the given cluster.