Alert dependency checking
Various embodiments provide for alert generation based on alert dependency. For some embodiments, the alert dependency checking facilitates alert noise reduction. Various embodiments described herein dynamically find or discover alert dependencies based on one or more alerts currently active, one or more active alerts generated in the past, or some combination of both. Various embodiments described herein provide alert monitoring that adapts based on an alert state of a machine. Various embodiments described herein generate a health score for a machine based on an alert state of the machine. Various embodiments described herein provide a tool for managing definitions of one or more alerts that can be identified as an active alert for a machine.
1. A method comprising:
accessing, by one or more hardware processors, telemetry data associated with a computing node the computing node operating one or more software services that can trigger one or more alerts;
evaluating, by the one or more hardware processors, the telemetry data to identify a set of triggered alerts triggered by the computing node, the telemetry data comprising log data generated by the computing node, the evaluating the telemetry data comprising:
obtaining, from the log data, alert data for a set of identifiable alerts; and
generating time-series alert data based on the alert data to identify the set of triggered alerts, the generating the time-series alert data comprising:
for a given identifiable alert described in the alert data, recording a time for each time a given metric, associated with the given identifiable alert, surpasses a threshold value;
accessing, by the one or more hardware processors, alert dependency data that describes a set of dependencies between a plurality of alerts; and
filtering, by the one or more hardware processors, the set of triggered alerts based on the alert dependency data to generate a set of active alerts.
2. The method of claim 1 , wherein the filtering the set of triggered alerts based on the alert dependency data to generate the set of active alerts comprises:
determining, based on the alert dependency data; whether a first triggered alert in the set of triggered alerts is dependent upon any other triggered alert in the set of triggered alerts; and
in response to determining that the first triggered alert is dependent upon a second triggered alert in the set of triggered alerts, generating the set of active alerts to exclude the first triggered alert.
3. The method of claim 1 , wherein the filtering the set of triggered alerts based on the alert dependency data to generate a set of active alerts comprises:
determining, based on the alert dependency data; whether a first triggered alert in the set of triggered alerts is dependent upon any other triggered alert in the set of triggered alerts; and
in response to determining that the first triggered alert is not dependent upon any other triggered alert in the set of triggered alerts, generating the set of active alerts to include the first triggered alert.
4. The method of claim 1 , wherein the plurality of alerts comprises a first alert and a second alert, and the set of dependencies comprises a particular dependency that describes that an occurrence of the first alert is dependent upon an occurrence of the second alert.
5. The method of claim 1 , wherein the alert dependency data is generated based on historical active alert data or correlation data between at least two alerts.
6. The method of claim 1 , wherein the generating the time-series alert data based on the alert further comprises:
for an individual identifiable alert described in the alert data, in response to each time an individual metric, associated with the individual identifiable alert, surpasses the threshold value, recording an amount by which the individual metric surpasses the threshold value.
7. The method of claim 1 , wherein the telemetry data comprises at least one of metric data regarding operation of the computing node or log data generated by the computing node.
8. The method of claim 1 , further comprising storing, by the one or more hardware processors, the set of active alerts as historical active alert data.
9. The method of claim 1 , further comprising performing, by the one or more hardware processors, alert dependency analysis, based on the set of active alerts and historical active alert data, to update the alert dependency data.
10. The method of claim 1 , wherein the alert dependency data is generated based on user-provided dependency mapping between two or more alerts.
11. The method of claim 1 , wherein the evaluating the telemetry data to identify the set of triggered alerts for the computing node comprises:
performing a set of searches with respect to the telemetry data, the set of searches corresponding to a set of identifiable alerts.
12. A system comprising:
a memory storing instructions; and
one or more hardware processors communicatively coupled to the memory and configured by the instructions to perform operations comprising:
accessing telemetry data associated with a computing node, the computing node operating one or more software services that can trigger one or more alerts;
evaluating the telemetry data to identify a set of triggered alerts triggered by the computing node the telemetry data comprising log data generated by the computing node, the evaluating the telemetry data comprising:
obtaining, from the log data, alert data for a set of identifiable alerts; and
generating time-series alert data based on the alert data to identify the set of triggered alerts, the generating the time-series alert data comprising:
for a given identifiable alert described in the alert data, recording a time for each time a given metric, associated with the given identifiable alert, surpasses a threshold value;
accessing alert dependency data that describes a set of dependencies between a plurality of alerts; and
filtering the set of triggered alerts based on the alert dependency data to generate a set of active alerts.
13. The system of claim 12 , wherein the operations further comprise;
storing the set of active alerts as historical active alert data.
14. The system of claim 12 , wherein the operations further comprise;
performing alert dependency analysis, based on the set of active alerts and historical active alert data, to update the alert dependency data.
15. The system of claim 12 , wherein the filtering the set of triggered alerts based on the alert dependency data to generate the set of active alerts comprises:
determining, based on the alert dependency data; whether a first triggered alert in the set of triggered alerts is dependent upon any other triggered alert in the set of triggered alerts; and
in response to determining that the first triggered alert is dependent upon a second triggered alert in the set of triggered alerts, generating the set of active alerts to exclude the first triggered alert.
16. The system of claim 12 , wherein the filtering the set of triggered alerts based on the alert dependency data to generate the set of active alerts comprises:
determining, based on the alert dependency data; whether a first triggered alert in the set of triggered alerts is dependent upon any other triggered alert in the set of triggered alerts; and
in response to determining that the first triggered alert is not dependent upon any other triggered alert in the set of triggered alerts, generating the set of active alerts to include the first triggered alert.
17. The system of claim 12 , wherein the plurality of alerts comprises a first alert and a second alert, and the set of dependencies comprises a particular dependency that describes that an occurrence of the first alert is dependent upon an occurrence of the second alert.
18. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising:
accessing telemetry data associated with a computing node, the computing node operating one or more software services that can trigger one or more alerts;
evaluating the telemetry data to identify a set of triggered alerts triggered by the computing node, the telemetry data comprising log data generated by the computing node, the evaluating the telemetry data comprising:
obtaining, from the log data, alert data for a set of identifiable alerts; and
generating time-series alert data based on the alert data to identify the set of triggered alerts, the generating the time-series alert data comprising:
for a given identifiable alert described in the alert data, recording a time for each time a given metric, associated with the given identifiable alert, surpasses a threshold value;
accessing alert dependency data that describes a set of dependencies between a plurality of alerts; and
filtering the set of triggered alerts based on the alert dependency data to generate a set of active alerts.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the operations further comprise:
storing the set of active alerts as historical active alert data.
20. The non-transitory computer-readable storage medium of claim 18 , wherein the operations further comprise:
performing alert dependency analysis, based on the set of active alerts and historical active alert data, to update the alert dependency data.