Detecting and mitigating cross-layer impact of change events on a cloud computing system
The present disclosure relates to systems, methods, and computer-readable media for identifying anomalies of failure events on a cloud computing system and determining cross-component and cross-layer correlation between change events that occur on the cloud computing system and the failure events associated with the anomalies. In particular, this disclosure describes a system that receives telemetry related to change events and failure events across any number of computing layers of a distributed computing environment (e.g., a cloud computing system) and detects anomalies based on counts of failure events that are manifested over discrete periods of time. Based on these detected anomalies, the anomaly correlation system can determine cross-layer and cross-component correlations between selective change events and the detected anomalies of failure events. The anomaly correlation system may further generate and provide recommendations related to mitigating or otherwise addressing the anomalies based on the determined correlations.
1 . A method, comprising:
receiving a plurality of change event identifiers associated with a plurality of change events detected across multiple hierarchical layers of a hierarchical structure of a cloud computing system;
maintaining a record of failure events across the multiple hierarchical layers of the cloud computing system indicating counts of detected failure events;
identifying a failure anomaly within the plurality of change events based on a count of detected failure events within the record of failure events exceeding a threshold count of failure events of an associated failure type;
determining whether the cloud computing system has a built-in remedy for addressing the failure anomaly; and
at least one of:
responsive to determining that the cloud computing system has a built-in remedy for addressing the failure anomaly, causing the built-in remedy to be executed without generating a recommendation including a recommended action to perform on one or more computing nodes of the cloud computing system; or
responsive to determining that the cloud computing system does not have a built-in remedy for addressing the failure anomaly:
selectively analyzing a subset of change event identifiers from the plurality of change event identifiers associated with the failure anomaly to determine a correlation between at least one change event and the failure anomaly, the subset of change event identifiers being associated with change events detected across at least two of the multiple hierarchical layers of the hierarchical structure of the cloud computing system; and
generating a recommendation including a recommended action to perform on one or more nodes of the cloud computing system associated with the at least one change event based on the determined correlation between the at least one change event and the failure anomaly.
2 . The method of claim 1 , wherein the plurality of change events includes a plurality of component upgrades on cloud components implemented across the multiple hierarchical layers of the cloud computing system.
3 . The method of claim 1 , wherein the multiple hierarchical layers include two different computing locations, the two different computing locations including two or more of:
a computing region;
a computing zone;
a node cluster;
a server rack; and
a server node.
4 . The method of claim 1 , wherein a change event identifier from the plurality of change event identifiers includes an identification of a change event detected by a cloud component of the cloud computing system and an indication of a layer from the multiple hierarchical layers where the cloud component is implemented on the cloud computing system.
5 . The method of claim 1 , wherein the record of failure events includes a table of rows, each row from the table including a failure identifier and a count of failure events associated with the failure identifier.
6 . The method of claim 5 , wherein each row further includes an identifier of a layer from the multiple hierarchical layers to which an associated failure event is associated.
7 . The method of claim 1 , wherein the threshold count is a threshold sigma value indicating a threshold standard deviation relative to an expected count of failures for an associated failure event.
8 . The method of claim 7 , wherein the expected count of failures is based on a combination of the associated failure type and a computing layer associated with the count of detected failures from the record of failure events.
9 . The method of claim 1 , wherein the at least one change event and the failure anomaly are each associated with respective components implemented across different computing layers of the cloud computing system.
10 . The method of claim 1 , wherein the recommendation includes an identification of one or more change events predicted to cause a plurality of detected failures associated with the count of failures exceeding the threshold count of failures.
11 . A method, comprising:
receiving a plurality of change event identifiers associated with a plurality of change events detected across multiple hierarchical layers of a hierarchical structure of a distributed computing system;
maintaining a record of failure events across the multiple hierarchical layers of the distributed computing system indicating counts of detected failure events;
identifying a failure anomaly within the plurality of change events based on a count of detected failure events within the record of failure events exceeding a threshold count of failure events of an associated failure type;
determining whether the distributed computing system has a built-in remedy for addressing the failure anomaly; and
at least one of:
responsive to determining that the distributed computing system has a built-in remedy for addressing the failure anomaly, causing the built-in remedy to be executed without generating a recommendation including a recommended action to perform on one or more computing nodes of the distributed computing system; or
responsive to determining that the distributed computing system does not have a built-in remedy for addressing the failure anomaly:
selectively analyzing a subset of change event identifiers from the plurality of change event identifiers associated with the failure anomaly to determine a correlation between at least one change event and the failure anomaly, the subset of change event identifiers being associated with change events detected across at least two of the multiple hierarchical layers of the hierarchical structure of the distributed computing system; and
generating a recommendation including a recommended action to perform on one or more nodes of the distributed computing system associated with the at least one change event based on the determined correlation between the at least one change event and the failure anomaly.
12 . The method of claim 11 , wherein the multiple hierarchical layers include two different computing locations, the two different computing locations including two or more of:
a computing region;
a computing zone;
a node cluster;
a server rack; and
a server node.
13 . The method of claim 11 , wherein the threshold count is a threshold sigma value indicating a threshold standard deviation relative to an expected count of failures for an associated failure event.
14 . A system, comprising:
at least one processor;
memory in electronic communication with the at least one processor; and
instructions stored in the memory, the instructions being executable by the at least one processor to:
receive a plurality of change event identifiers associated with a plurality of change events detected across multiple hierarchical layers of a hierarchical structure of a cloud computing system;
maintain a record of failure events across the multiple hierarchical layers of the cloud computing system indicating counts of detected failure events;
identify a failure anomaly within the plurality of change events based on a count of detected failure events within the record of failure events exceeding a threshold count of failure events of an associated failure type;
determine whether the cloud computing system has a built-in remedy for addressing the failure anomaly; and
at least one of:
responsive to determining that the cloud computing system has a built-in remedy for addressing the failure anomaly, causing the built-in remedy to be executed without generating a recommendation including a recommended action to perform on one or more computing nodes of the cloud computing system; or
responsive to determining that the cloud computing system does not have a built-in remedy for addressing the failure anomaly
selectively analyze a subset of change event identifiers from the plurality of change event identifiers associated with the failure anomaly to determine a correlation between at least one change event and the failure anomaly, the subset of change event identifiers being associated with change events detected across at least two of the multiple hierarchical layers of the hierarchical structure of the cloud computing system; and
generate a recommendation including a recommended action to perform on one or more nodes of the cloud computing system associated with the at least one change event based on the determined correlation between the at least one change event and the failure anomaly and generate a recommendation based on the determined correlation between the at least one change event and the failure anomaly.
15 . The system of claim 14 , wherein the plurality of change events includes a plurality of component upgrades on cloud components implemented across multiple hierarchical layers of the hierarchical structure of the cloud computing system.
16 . The system of claim 14 , wherein the multiple hierarchical layers include two different computing locations, the two different computing locations including two or more of:
a computing region;
a computing zone;
a node cluster;
a server rack; and
a server node.
17 . The system of claim 14 , wherein the threshold count is a threshold sigma value indicating a threshold standard deviation relative to an expected count of failures for an associated failure event.