Large-scale distributed correlation

1. A system, comprising:

one or more computing devices; and

a local-level node, implemented on the one or more computing devices, configured to:

trigger an alert for a performance metric of an application executing on the local-level node, wherein the alert indicates anomalous behavior for the performance metric, and

send the alert to a higher-level node, implemented on the one or more computing devices;

receive a distributed correlation request from the higher-level node, wherein the distributed correlation request is initiated to determine a root cause of the alert;

construct a correlation graph, the correlation graph including a root node representing the performance metric, a plurality of leaf nodes representing other performance metrics correlated with the performance metric, and a plurality of edges connecting the root node and the plurality of leaf nodes, each edge representing a dependent relationship between two performance metrics;

assign a correlation strength to each of the plurality of edges;

select one or more of the plurality of leaf nodes to be included in a correlation result based on the correlation strength assigned to each of the plurality of edges connected to the plurality of leaf nodes; and

send the correlation result to the higher-level node;

wherein the higher-level node is configured to:

select a probable cause of triggering the alert based on the performance metrics represented by the one or more leaf nodes included in the correlation result; and

present the probable cause to a user.

2. The system of claim 1 , wherein to trigger the local-level node is further configured to:

detect that the performance metric contains metric values that are anomalous by calculating deviations of the metric values from a dynamic baseline.

3. The system of claim 1 , wherein the local-level node is further configured to:

collect performance metric values for the performance metric across a period of time;

consolidate the performance metric values, collected across the period of time, into a bucketized histogram time slice; and

send the consolidated performance metric values to the higher-level node.

4. The system of claim 3 , wherein the higher-level node has a parent higher-level node, and wherein the higher-level node is further configured to:

receive the distributed correlation request from the parent higher-level node, wherein the distributed correlation request includes the performance metric;

perform correlation between the performance metric and other metrics collected or received at the higher-level node, wherein the correlation is performed using the consolidated performance metric values; and

select the probable cause based on the performed correlation and the correlation results.

5. The system of claim 3 , wherein the local-level node is further configured to:

collect second performance metric values for a second performance metric across the period of time;

consolidate the second performance metric values into a second bucketized histogram time slice; and

correlate the bucketized histogram time slice against the second bucketized histogram time slice in order to produce the correlation result.

6. The system of claim 1 , wherein the local-level node is further configured to:

track application-dependency-mapping information for the application; and

send the application-dependency-mapping information to the higher-level node.

7. The system of claim 6 , wherein the higher-level node is further configured to:

receive the application-dependency-mapping information from the local-level node; and

send the distributed correlation request to the lower-level node depending on whether the received application-dependency-mapping information indicates a relationship between the higher-level node and the local-level node.

8. The system of claim 6 , wherein the local-level node is further configured to:

prune leaf nodes of the correlation graph depending on whether the application-dependency-mapping information conflicts with the plurality of edges connecting the root node and the plurality of leaf nodes.

9. A method, comprising:

triggering, by a local-level node that is implemented on one or more computing devices, an alert for a performance metric of an application executing on the local-level node, wherein the alert indicates anomalous behavior for the performance metric;

sending the alert to a higher-level node that is implemented on the one or more computing devices;

receiving a distributed correlation request from the higher-level node, wherein the distributed correlation request is initiated to determine a root cause of the alert;

constructing a correlation graph, the correlation graph including a root node representing the performance metric, a plurality of leaf nodes representing other performance metrics correlated with the performance metric, and a plurality of edges connecting the root node and the plurality of leaf nodes, each edge representing a dependent relationship between two performance metrics;

assigning a correlation strength to each of the plurality of edges;

selecting one or more of the plurality of leaf nodes to be included in a correlation result based on the correlation strength assigned to each of the plurality of edges connected to the plurality of leaf nodes; and

sending the correlation result to the higher-level node, wherein the higher-level node is configured to select a probable cause of triggering the alert based on the performance metrics represented by the one or more leaf nodes included in the correlation result, and present the probable cause to a user.

10. The method of claim 9 , wherein the triggering further comprises:

detecting, at the local-level node, if the performance metric contains metric values that are anomalous by calculating deviations of the metric values from a dynamic baseline.

11. The method of claim 9 , further comprising:

collecting, at the local-level node, performance metric values for the performance metric across a period of time;

consolidating the performance metric values, collected across the period of time, into

a bucketized histogram time slice;

and

sending the consolidated performance metric values to the higher-level node.

12. The method of claim 11 , further comprising:

collecting, at the local-level node, second performance metric values for a second performance metric across the period of time;

consolidating the second performance metric values into a second bucketized histogram time slice; and

correlating the bucketized histogram time slice against the second bucketized histogram time slice in order to produce the correlation result.

13. The method of claim 9 , wherein the higher-level node has a parent higher-level node, the method further comprising:

receiving by the higher level node the distributed correlation request from the parent higher-level node, wherein the distributed correlation request includes the performance metric;

performing by the higher level node correlation between the performance metric and other metrics collected or received at the higher-level node, wherein the correlation is performed using the consolidated performance metric values; and

selecting by the higher level node the probable cause based on the performed correlation and the correlation results.

14. The method of claim 9 , further comprising:

tracking application-dependency-mapping information for the application; and

sending the application-dependency-mapping information to the higher-level node.

15. The method of claim 14 , further comprising:

receiving, at the lower-level node, the distributed correlation request depending on whether the application dependency mapping information indicates a relationship between the higher-level node and the lower-level node.

16. The method of claim 14 , further comprising:

pruning leaf nodes of the correlation graph depending on whether the received application-dependency-mapping information conflicts with the plurality of edges connecting the root node and the plurality of leaf nodes;

selecting a remaining leaf node, associated with a highly correlated metric, to be included in the correlation result.

17. A non-transitory computer readable storage medium having instructions stored thereon that, in response to execution by a computing device, cause the computing device to perform operations for performing distributed correlation in order to diagnose a root cause of anomalous behavior for a performance metric, the operations comprising:

triggering, by a local-level node that is implemented on one or more computing devices, an alert for a performance metric of an application executing on the local-level node, wherein the alert indicates anomalous behavior for the performance metric;

sending the alert to a higher-level node that is implemented on the one or more computing devices;

receiving a distributed correlation request from the higher-level node, wherein the distributed correlation request is initiated to determine a root cause of the alert;

constructing a correlation graph, the correlation graph including a root node representing the performance metric, a plurality of leaf nodes representing other performance metrics correlated with the performance metric, and a plurality of edges connecting the root node and the plurality of leaf nodes, each edge representing a dependent relationship between two performance metrics;

assigning a correlation strength to each of the plurality of edges;

selecting one or more of the plurality of leaf nodes to be included in a correlation result based on the correlation strength assigned to each of the plurality of edges connected to the plurality of leaf nodes; and

sending the correlation result to the higher-level node, wherein the higher-level node is configured to select a probable cause of triggering the alert based on the performance metrics represented by the one or more leaf nodes included in the correlation result, and present the probable cause to a user.