Failure detection in cloud-computing systems
Techniques discussed herein monitoring for and identifying failures in a cloud-computing environment. Proxy devices can be communicatively disposed between services of the cloud-computing environment. The proxy devices can capture call stack data corresponding to function calls between services. A directional graph can be generated based on that call stack data that represents the communication paths between cloud-computing resources (e.g., the services). Ingress paths to a service can be evaluated by calculating various metrics for each path. Using these metrics, failures can be isolated to a particular communication path and/or a particular, and potentially relatively distant, upstream service.
1. A computer-implemented method, comprising:
capturing call stack data corresponding to a plurality of cloud-computing resources, the call stack data being captured by a plurality of proxy devices, the plurality of proxy devices being distinct from the cloud-computing resources, each respective proxy device being configured to monitor communications between a respective set of pairs of cloud-computing resources of the plurality of cloud-computing resources, and the set of pairs of cloud-computing resources associated with a particular proxy device being different from the sets of pairs of cloud-computing resources associated with other proxy devices of the plurality of proxy devices;
generating a directional graph comprising a plurality of nodes and a plurality of edges, each of the plurality of nodes representing a respective cloud-computing resource of the plurality of cloud-computing resources, and each edge of the plurality of edges representing a communications path between a respective pair of cloud-computing resources of the plurality of cloud-computing resources through a proxy device of the plurality of proxy devices;
identifying a plurality of ingress communication paths to a cloud-computing resource based at least in part on identifying, from the directional graph, a set of edges of the plurality of edges that end at a particular node of the plurality of nodes that represents the cloud-computing resource;
calculating a plurality of metrics corresponding to each of the plurality of ingress communication paths identified with the set of edges ending at the particular node, the plurality of metrics being calculated based at least in part on the call stack data captured by proxy devices that are configured to monitor communications over the plurality of ingress communication paths;
identifying, based at least in part on the plurality of metrics, that a number of respective failures exist over the plurality of ingress communication paths;
identifying that the number of respective failures identified for the cloud-computing resource exceeds a threshold with respect to a count of the plurality of ingress communication paths to the cloud-computing resource;
responsive to identifying that the number of respective failures exceeds the threshold, identifying that a failure exists at the cloud-computing resource; and
providing a notification indicating existence of the failure.
2. The computer-implemented method of claim 1 , further comprising:
identifying that two or more edges of the directional graph collectively define an aggregate path from a first cloud-computing resource to a second cloud-computing resource, the second cloud-computing resource being communicatively connected to the first cloud-computing resource via one or more intervening cloud-computing resources;
calculating a collective latency value based at least in part on a respective latency value corresponding to each edge in the aggregate path;
identifying that the collective latency value exceeds a predefined threshold value; and
identifying the aggregate path as failure based at least in part on identifying that the collective latency value exceeds the corresponding predefined threshold value.
3. The computer-implemented method of claim 1 , wherein each instance of the call stack data comprising a source identifier of a first cloud-computing resource of the plurality of cloud-computing resources, a destination identifier for a second cloud-computing resource of the plurality of cloud-computing resources, a request identifier, a latency value, and a response code.
4. The computer-implemented method of claim 1 , wherein calculating the plurality of metrics comprises calculating i) a latency value and ii) a success rate value for each of the plurality of ingress communication paths to the cloud-computing resource, the latency value and the success rate value being calculated based at least in part on obtaining corresponding call stack data that corresponds to a time period, the corresponding call stack data being associated with the plurality of ingress communication paths to the cloud-computing resource.
5. The computer-implemented method of claim 1 , wherein identifying that the failure exists comprises at least one of:
determining that a latency value corresponding to one or more ingress communications paths of the plurality of ingress communication paths exceeds a latency threshold; or
determining that a success rate value corresponding to one or more ingress communications paths of the plurality of ingress communication paths exceeds a success rate threshold.
6. The computer-implemented method of claim 1 , wherein the call stack data with which the plurality of metrics corresponding to each of the plurality of ingress communication paths are calculated is selected from a corpus of call stack data based at least in part on identifying the call stack data is associated with a particular time period.
7. A cloud-computing system, comprising:
one or more processors; and
one or more memories storing computer-executable instructions that, when executed with the one or more processors, cause the cloud-computing system to:
capture call stack data corresponding to a plurality of cloud-computing resources, the call stack data being captured by a plurality of proxy devices, the plurality of proxy devices being distinct from the cloud-computing resources, each respective proxy device being configured to monitor communications between a respective set of pairs of cloud-computing resources of the plurality of cloud-computing resources, and the set of pairs of cloud-computing resources associated with a particular proxy device being different from the sets of pairs of cloud-computing resources associated with other proxy devices of the plurality of proxy devices;
generate a directional graph comprising a plurality of nodes and a plurality of edges, representing a respective cloud-computing resource of the plurality of cloud-computing resources, and each edge of the plurality of edges representing a communications path between a respective pair of cloud-computing resources of the plurality of cloud-computing resources through a proxy device of the plurality of proxy devices;
identify a plurality of ingress communication paths to a cloud-computing resource based at least in part on identifying, from the directional graph, a set of edges of the plurality of edges that end at a particular node of the plurality of nodes that represents the cloud-computing resource;
calculate a plurality of metrics corresponding to each of the plurality of ingress communication paths identified with the set of edges ending at the particular node, the plurality of metrics being calculated based at least in part on the call stack data captured by the proxy devices that are configured to monitor communications over the plurality of ingress communication paths;
identify, based at least in part on the plurality of metrics, that a number of respective failures exist over the plurality of ingress communication paths;
identify that the number of respective failures exceeds a threshold with respect to a count of the plurality of ingress communication paths to the cloud-computing resource;
responsive to identifying that the number of respective failures exceeds the threshold, identify that a failure exists at the cloud-computing resource; and
provide a notification indicating existence of the failure.
8. The cloud-computing system of claim 7 , wherein executing the instructions further causes the cloud-computing system to:
identify that two or more edges of the directional graph collectively define an aggregate path from a first cloud-computing resource to a second cloud-computing resource, the second cloud-computing resource being communicatively connected to the first cloud-computing resource via one or more intervening cloud-computing resources;
calculate a collective latency value based at least in part on a respective latency value corresponding to each edge in the aggregate path;
identify that the collective latency value exceeds a predefined threshold value; and
identify the aggregate path as a failure based at least in part on identifying that the collective latency value exceeds the corresponding predefined threshold value.
9. The cloud-computing system of claim 7 , wherein each instance of the call stack data comprising a source identifier of a first cloud-computing resource of the plurality of cloud-computing resources, a destination identifier for a second cloud-computing resource of the plurality of cloud-computing resources, a request identifier, a latency value, and a response code.
10. The cloud-computing system of claim 7 , wherein calculating the plurality of metrics comprises calculating i) a latency value and ii) a success rate value for each of the plurality of ingress communication paths to the cloud-computing resource, the latency value and the success rate value being calculated based at least in part on obtaining corresponding call stack data that corresponds to a time period, the corresponding call stack data being associated with the plurality of ingress communication paths to the cloud-computing resource.
11. The cloud-computing system of claim 7 , wherein identifying that the failure exists causes the cloud-computing system to:
determine that a latency value corresponding to one or more ingress communications paths of the plurality of ingress communication paths exceeds a latency threshold; or
determine that a success rate value corresponding to one or more ingress communications paths of the plurality of ingress communication paths exceeds a success rate threshold.
12. The cloud-computing system of claim 7 , wherein the call stack data with which the plurality of metrics corresponding to each of the plurality of ingress communication paths are calculated is selected from a corpus of call stack data based at least in part on identifying the call stack data is associated with a particular time period.
13. A non-transitory computer-readable medium comprising executable instructions that, when executed with one or more processors of a cloud-computing system, cause the cloud-computing system to:
capture call stack data corresponding to a plurality of cloud-computing resources, the call stack data being captured by a plurality of proxy devices, the plurality of proxy devices being distinct from the cloud-computing resources, each respective proxy device being configured to monitor communications between a respective set of pairs of cloud-computing resources of the plurality of cloud-computing resources, and the set of pairs of cloud-computing resources associated with a particular proxy device being different from the sets of pairs of cloud-computing resources associated with other proxy devices of the plurality of proxy devices;
generate a directional graph comprising a plurality of nodes and a plurality of edges, each of the plurality of nodes representing a respective cloud-computing resource of the plurality of cloud-computing resources, and each edge of the plurality of edges representing a communications path between a respective pair of cloud-computing resources of the plurality of cloud-computing resources through a proxy device of the plurality of proxy devices;
identify a plurality of ingress communication paths to a cloud-computing resource based at least in part on identifying, from the directional graph, a set of edges of the plurality of edges that end at a particular node of the plurality of nodes that represents the cloud-computing resource;
calculate a plurality of metrics corresponding to each of the plurality of ingress communication paths identified with the set of edges ending at the particular node, the plurality of metrics being calculated based at least in part on the call stack data captured by the proxy devices that are configured to monitor communications over the plurality of ingress communication paths;
identify, based at least in part on the plurality of metrics, that a number of respective failures exist over the plurality of ingress communication paths;
identify that the number of respective failures identified for the cloud-computing resource exceeds a threshold with respect to a count of the plurality of ingress communication paths to the cloud-computing resource;
responsive to identifying that the number of respective failures exceeds the threshold, identify that a failure exists at the cloud-computing resource; and
provide a notification indicating existence of the failure.
14. The non-transitory computer-readable medium of claim 13 , wherein executing the instructions further causes the cloud-computing system to:
identify that two or more edges of the directional graph collectively define an aggregate path from a first cloud-computing resource to a second cloud-computing resource, the second cloud-computing resource being communicatively connected to the first cloud-computing resource via one or more intervening cloud-computing resources;
calculate a collective latency value based at least in part on a respective latency value corresponding to each edge in the aggregate path;
identify that the collective latency value exceeds a predefined threshold value; and
identify the aggregate path as a failure based at least in part on identifying that the collective latency value exceeds the corresponding predefined threshold value.
15. The non-transitory computer-readable medium of claim 13 , wherein each instance of the call stack data comprising a source identifier of a first cloud-computing resource of the plurality of cloud-computing resources, a destination identifier for a second cloud-computing resource of the plurality of cloud-computing resources, a request identifier, a latency value, and a response code.
16. The non-transitory computer-readable medium of claim 13 , wherein calculating the plurality of metrics comprises calculating i) a latency value and ii) a success rate value for each of the plurality of ingress communication paths to the cloud-computing resource, the latency value and the success rate value being calculated based at least in part on obtaining call stack data from a time period, the call stack data being associated with the plurality of ingress communication paths to the cloud-computing resource.
17. The non-transitory computer-readable medium of claim 13 , wherein identifying that the failure exists causes the cloud-computing system to:
determine that a latency value corresponding to one or more ingress communications paths of the plurality of ingress communication paths exceeds a latency threshold; or
determine that a success rate value corresponding to one or more ingress communications paths of the plurality of ingress communication paths exceeds a success rate threshold.