Machine learning for metric collection
A performance monitoring system includes a metric collector configured to receive, via metric exporters, telemetry data comprising metrics related to a network of computing devices. A metric time series database stores related metrics. An alert rule evaluator service is configured to evaluate rules using stored metrics. The performance monitoring system may include a machine learning module and is configured to determine optimized metric collection sampling intervals and rule evaluation intervals, and to automatically determine recommended alert rules.
1. A method comprising:
collecting, by a performance monitoring system, telemetry data comprising metrics related to a network of computing devices, wherein, for each metric, metric values associated with a corresponding metric name of the metric are collected at each of a plurality of times according to a first collection sampling interval;
evaluating, by the performance monitoring system, alert rules using the telemetry data, including evaluating a first alert rule of the alert rules by comparing metric values associated with a corresponding metric name of a first metric to a corresponding threshold value of the first alert rule at a plurality evaluation times;
generating a metric attribute related to the alert rules based on a count related to the evaluating the first alert rule;
determining, by the performance monitoring system, a predicted metric weight for the first metric based on the metric attribute; and
determining, by the performance monitoring system, a second collection sampling interval for the first metric based on the predicted metric weight.
2. The method of claim 1 , wherein the second collection sampling interval is different than the first collection sampling interval, the method further comprising subsequently collecting, by the performance monitoring system, corresponding metric values for the first metric using the second collection sampling interval.
3. The method of claim 1 , further comprising determining, by the performance monitoring system, whether the first metric is a critical metric, and if the first metric is the critical metric, using a predetermined minimum sampling interval for the second collection sampling interval.
4. The method of claim 1 , wherein collecting the telemetry data comprises performing a pull based collection process to collect the telemetry data.
5. The method of claim 1 , wherein determining the predicted metric weight comprises performing a machine learning module to determine the predicted metric weight.
6. The method of claim 1 , wherein determining the predicted metric weight comprises performing a regression analysis to determine the predicted metric weight.
7. The method of claim 1 , further comprising updating a current collection sampling interval for the first metric on an on-going basis.
8. The method of claim 1 , further comprising determining one or more related metrics that are related to the first metric, wherein the metric attribute includes a first metric access ratio and a group metrics access ratio for the one or more related metrics.
9. The method of claim 1 , wherein the metric attribute is related to an access rate of a metric for the alert rules, and wherein the count is a count of times a metric is accessed in a fixed time duration.
10. The method of claim 1 , wherein the metric attribute is a metric access ratio which is a ratio of access rate to a sample rate for the metric.
11. The method of claim 1 , wherein the metric attribute is related to a hit rate of a metric for the alert rules, and wherein the count is a number of times the metric values associated with the corresponding metric name of the first metric is greater than the corresponding threshold value of the first alert rule.
12. The method of claim 1 , further comprising instructing metric exporters in the network to provide the first metric according to the second collection sampling interval.
13. A performance monitoring system, comprising:
a memory; and
one or more processors in communication with the memory, the one or more processors configured to execute a collector and an alert rule evaluator service,
wherein the collector is configured to receive telemetry data via metric exporters, the telemetry data comprising metrics related to a network of computing devices, wherein, for each metric, metric values associated with a corresponding metric name are configured to be collected at each of a plurality of times based on a corresponding first predetermined collection sampling interval;
wherein the alert rule evaluator service is configured to evaluate alert rules using the telemetry data, wherein, to evaluate a first alert rule of the alert rules, the alert rule evaluator service compares metric values associated with a corresponding metric name of a first metric, compares a corresponding metric value to a corresponding threshold value of the first alert rule at each of a plurality of rule evaluation times, generates an alert if a comparison condition of the first alert rule is met, and stores the generated alerts in the memory,
wherein the collector is configured to: access the generated alerts in the memory to calculate a metric attribute of the first metric related to the alert rules based on a count related to the evaluating the first alert rule, determine a predicted metric weight using the metric attribute of the first metric, and determine a second collection sampling interval for the first metric based on the predicted metric weight.
14. The performance monitoring system of claim 13 , wherein the collector is configured to subsequently collect metric values of the first metric using the second collection sampling interval.
15. The performance monitoring system of claim 13 , wherein the collector is configured to determine whether the first metric is a critical metric, and if the first metric is the critical metric, a predetermined minimum sampling interval is used for the second collection sampling interval.
16. The performance monitoring system of claim 13 , wherein the collector further comprises a machine learning module configured to determine the predicted metric weight.
17. The performance monitoring system of claim 13 , wherein the collector is configured to perform a regression analysis to determine the predicted metric weight.
18. The performance monitoring system of claim 13 , wherein the collector is configured to update a current collection sampling interval for the first metric on an on-going basis.
19. The performance monitoring system of claim 13 , wherein the collector includes a critical metrics discovery service for determining one or more related metrics that are related to the first metric and wherein the metric attribute is a first metric access ratio and a group metrics access ratio for the one or more related metrics.
20. The performance monitoring system of claim 13 , further comprising:
a metric time series database and a query history database, wherein the one or more processors are further configured to execute a metric querier in communication with the alert rule evaluator service and the metric time series database, the metric querier enabled to receive user queries regarding a metric and store user queries in the query history database, wherein the collector accesses the query history database to determine the predicted metric weight and the second collection sampling interval.