MACHINE LEARNING FOR RULE RECOMMENDATION
A performance monitoring system includes a metric collector configured to receive, via metric exporters, telemetry data comprising metrics related to a network of computing devices. A metric time series database stores related metrics. An alert rule evaluator service is configured to evaluate rules using stored metrics. The performance monitoring system may include a machine learning module and is configured to automatically determine recommended alert rules.
1 . A method for recommending alert rules for a performance monitoring system, the method comprising:
receiving, by a performance monitoring system, user-created alert rules;
collecting, by the performance monitoring system, telemetry data including a plurality of metrics related to the user-created rules;
reading, by the performance monitoring system, a first alert rule of the user-created alert rules, wherein the first alert rule is associated with a first metric of the plurality of metrics;
determining, by the performance monitoring system, at least one related metric that is related to the first metric;
creating, by the performance monitoring system, a set of provisional relevant rules using the at least one related metric;
evaluating, by the performance monitoring system using the telemetry data, each provisional relevant rule in the set of provisional relevant rules to determine a corresponding provisional rule evaluation attribute;
determining, by the performance monitoring system, for each provisional relevant rule in the set of provisional relevant rules, a corresponding relevant rule weight based on the corresponding provisional rule evaluation attribute; and
determining, by the performance monitoring system, whether each provisional relevant rule is a recommended relevant rule based on the corresponding relevant rule weight.
2 . The method of claim 1 , wherein determining whether each provisional relevant rule is a recommended relevant rule comprises determining if a corresponding relevant rule weight is greater than a predetermined threshold weight.
3 . The method of claim 1 , wherein determining whether each provisional relevant rule is an recommended relevant rule comprises determining if a corresponding relevant rule weight is less than a predetermined acceptable threshold weight to determine that the provisional relevant rule is irrelevant.
4 . The method of claim 1 , wherein a corresponding provisional rule evaluation attribute for a provisional relevant rule is at least one of a corresponding provisional rule hit rate and a corresponding provisional rule miss rate.
5 . The method of claim 4 , further comprising evaluating the first alert rule using the telemetry data to determine at least one of a first alert rule hit rate and a first alert rule miss rate and using the at least one of a first alert rule hit rate and a first alert rule miss rate to determine a corresponding provisional rule evaluation attribute.
6 . The method of claim 1 , wherein a corresponding provisional rule evaluation attribute for a provisional relevant rule includes a provisional rule hit rate and a provisional rule miss rate, the method further comprising evaluating the first alert rule using the telemetry data to determine a first alert rule hit rate and a first alert rule miss rate and using the first alert rule hit rate and the first alert rule miss rate to determine additional corresponding provisional rule evaluation attributes that are used in determining a corresponding provisional rule weight.
7 . The method of claim 6 , wherein the additional corresponding provisional rule evaluation attributes include a relative provisional rule hit rate and a relative provisional rule miss rate.
8 . The method of claim 1 , wherein creating a set of provisional relevant rules using the at least one related metric includes using at least one of metric aggregation and a set of comparison operators.
9 . The method of claim 1 , further comprising automatically implementing a recommended relevant rule.
10 . The method of claim 1 , further comprising providing a recommended relevant rule to a user for approval prior to implementation of the recommended relevant rule.
11 . The method of claim 1 , further comprising determining and evaluating a corresponding set of provisional rules for each of the user-created alert rules.
12 . The method of claim 1 , wherein determining a corresponding relevant rule weight based on the corresponding provisional rule evaluation attribute comprises using a machine learning model.
13 . The method of claim 1 , wherein determining, by the least one related metric that is related to the first metric comprises matching a source of the first metric or matching a label of the first metric.
14 . A performance monitoring system comprising processing circuitry coupled to a memory device, the memory and processing circuitry configured to:
receive user-created alert rules;
collect telemetry data including a plurality of metrics related to the user-created rules;
read a first alert rule of the user-created alert rules, wherein the first alert rule is associated with a first metric of the plurality of metrics;
determine at least one related metric that is related to the first metric;
create a set of provisional relevant rules using the at least one related metric;
evaluate each provisional relevant rule in the set of provisional relevant rules using the telemetry data to determine a corresponding provisional rule evaluation attribute;
determine, for each provisional relevant rule in the set of provisional relevant rules, a corresponding relevant rule weight based on the corresponding provisional rule evaluation attribute; and
determine whether each provisional relevant rule is a recommended relevant rule or an irrelevant rule based on the corresponding relevant rule weight.
15 . The performance monitoring system of claim 14 , wherein the memory and processing circuitry are configured to determine whether a corresponding relevant rule weight is greater than a predetermined threshold weight.
16 . The performance monitoring system of claim 14 , wherein a corresponding provisional rule evaluation attribute for a provisional relevant rule is at least one of a corresponding provisional rule hit rate and a corresponding provisional rule miss rate.
17 . The performance monitoring system of claim 14 , wherein the memory and processing circuitry are configured to evaluate the first alert rule using the telemetry data to determine at least one of a first alert rule hit rate and a first alert rule miss rate and use the at least one of a first alert rule hit rate and a first alert rule miss rate to determine a corresponding provisional rule evaluation attribute.
18 . The performance monitoring system of claim 14 , wherein a corresponding provisional rule evaluation attribute for a provisional relevant rule includes a provisional rule hit rate and a provisional rule miss rate, wherein the memory and processing circuitry are configured to evaluate a first alert rule using the telemetry data to determine a first alert rule hit rate and a first alert rule miss rate and use the first alert rule hit rate and the first alert rule miss rate to determine additional corresponding provisional rule evaluation attributes that are used to determine a corresponding provisional rule weight.
19 . The performance monitoring system of claim 14 , wherein the memory and processing circuitry are configured to determine and evaluate a corresponding set of provisional rules for each of the user-created alert rules.
20 . A non-transitory computer-readable storage medium storing instructions thereon that when executed cause one or more processors, via execution of a performance monitoring system, to:
receive user-created alert rules;
collect telemetry data including a plurality of metrics related to the user-created rules;
read a first alert rule of the user-created alert rules, wherein the first alert rule is associated with a first metric of the plurality of metrics;
determine at least one related metric that is related to the first metric;
create a set of provisional relevant rules using the at least one related metric;
evaluate each provisional relevant rule in the set of provisional relevant rules using the telemetry data to determine a corresponding provisional rule evaluation attribute;
determine, for each provisional relevant rule in the set of provisional relevant rules, a corresponding relevant rule weight based on the corresponding provisional rule evaluation attribute; and
determine whether each provisional relevant rule is a recommended relevant rule or an irrelevant rule based on the corresponding relevant rule weight.