Machine learning for metric collection
A performance monitoring system includes a metric collector configured to receive, via metric exporters, telemetry data comprising metrics related to a network of computing devices. A metric time series database stores related metrics. An alert rule evaluator service is configured to evaluate rules using stored metrics. The performance monitoring system may include a machine learning module and is configured to determine optimized metric collection sampling intervals and rule evaluation intervals, and to automatically determine recommended alert rules.
1. A method comprising:
collecting, by a performance monitoring system, telemetry data comprising metrics related to a network of computing devices, wherein, for each metric, metric values associated with a corresponding metric name of the metric are collected at each of a plurality of times according to a first collection sampling interval;
storing, by the performance monitoring system, a query history comprising queries related to the metrics;
generating, by the performance monitoring system, a metric relevance attribute based on an access ratio determined from a portion of the query history corresponding to a first metric of the metrics;
determining, by the performance monitoring system, a predicted metric weight for the first metric based on the metric relevance attribute; and
collecting, by the performance monitoring system, additional telemetry data according to a second collection sampling interval for at least one of the metrics, wherein the second collection sampling interval is based on the predicted metric weight for the first metric.
2. The method of claim 1 ,
wherein the second collection sampling interval is different than the first collection sampling interval, and
wherein collecting the additional telemetry data comprises collecting corresponding metric values for the first metric according to the second collection sampling interval.
3. The method of claim 1 , further comprising:
determining, by the performance monitoring system, whether the first metric is a critical metric; and
based on determining the first metric is a critical metric, using, by the performance monitoring system, a predetermined minimum sampling interval for the second collection sampling interval.
4. The method of claim 1 , wherein collecting the telemetry data comprises performing, by the performance monitoring system, a pull based collection process to collect the telemetry data.
5. The method of claim 1 , wherein determining the predicted metric weight comprises applying, by the performance monitoring system, a machine learning model to determine the predicted metric weight.
6. The method of claim 1 , wherein determining the predicted metric weight comprises performing, by the performance monitoring system, a regression analysis to determine the predicted metric weight.
7. The method of claim 1 , further comprising updating, by the performance monitoring system, the second collection sampling interval for the first metric on an on-going basis.
8. The method of claim 1 , further comprising determining, by the performance monitoring system, one or more related metrics that are related to the first metric, wherein the metric relevance attribute is based on the access ratio and a group metrics access ratio for the one or more related metrics.
9. A performance monitoring system, comprising:
a memory that stores instructions; and
processing circuitry coupled to the memory that executes the instructions to:
collect telemetry data comprising metrics related to a network of computing devices, wherein, for each metric, metric values associated with a corresponding metric name of the metric are collected at each of a plurality of times according to a first collection sampling interval;
store a query history comprising queries related to the metrics;
generate a metric relevance attribute based on an access ratio determined from a portion of the query history corresponding to a first metric of the metrics;
determine a predicted metric weight for the first metric based on the metric relevance attribute; and
collect additional telemetry data according to a second collection sampling interval for at least one of the metrics, wherein the second collection sampling interval is based on the predicted metric weight for the first metric.
10. The system of claim 9 ,
wherein the second collection sampling interval is different than the first collection sampling interval, and
wherein to collect the additional telemetry data the processing circuitry executes the instructions to collect corresponding metric values for the first metric according to the second collection sampling interval.
11. The system of claim 9 , wherein the processing circuitry further executes the instructions to:
determine whether the first metric is a critical metric; and
based on determining the first metric is a critical metric, use a predetermined minimum sampling interval for the second collection sampling interval.
12. The system of claim 9 , wherein to collect the telemetry data the processing circuitry further executes the instructions to perform a pull based collection process to collect the telemetry data.
13. The system of claim 9 , wherein to determine the predicted metric weight the processing circuitry further executes the instructions to apply a machine learning model to determine the predicted metric weight.
14. The system of claim 9 , wherein to determine the predicted metric weight the processing circuitry further executes the instructions to perform a regression analysis to determine the predicted metric weight.
15. The system of claim 9 , wherein the processing circuitry further executes the instructions to update the second collection sampling interval for the first metric on an on-going basis.
16. The system of claim 9 , wherein the processing circuitry further executes the instructions to determine one or more related metrics that are related to the first metric, wherein the metric relevance attribute is based on the access ratio and a group metrics access ratio for the one or more related metrics.
17. Non-transitory computer-readable storage media comprising instructions, that when executed by processing circuitry of a computing system, cause the processing circuitry to:
collect telemetry data comprising metrics related to a network of computing devices, wherein, for each metric, metric values associated with a corresponding metric name of the metric are collected at each of a plurality of times according to a first collection sampling interval;
store a query history comprising queries related to the metrics;
generate a metric relevance attribute based on an access ratio determined from a portion of the query history corresponding to a first metric of the metrics;
determine a predicted metric weight for the first metric based on the metric relevance attribute; and
collect additional telemetry data according to a second collection sampling interval for at least one of the metrics, wherein the second collection sampling interval is based on the predicted metric weight for the first metric.
18. The non-transitory computer-readable storage media of claim 17 ,
wherein the second collection sampling interval is different than the first collection sampling interval, and
wherein to collect the additional telemetry data the instructions, when further executed by the processing circuitry, cause the processing circuitry to collect corresponding metric values for the first metric according to the second collection sampling interval.
19. The non-transitory computer-readable storage media of claim 17 , wherein the instructions, when further executed by the processing circuitry, cause the processing circuitry to:
determine whether the first metric is a critical metric; and
based on determining the first metric is a critical metric, use a predetermined minimum sampling interval for the second collection sampling interval.
20. The non-transitory computer-readable storage media of claim 17 , wherein the instructions, when further executed by the processing circuitry, cause the processing circuitry to determine one or more related metrics that are related to the first metric, wherein the metric relevance attribute is based on the access ratio and a group metric access ratio for the one or more related metrics.