Method and system for solving for seamless resiliency in a multi-tier network for grey failures
An automatically self-healing multi-tier system for providing seamless resiliency for end users is provided. The system includes a plurality of tiers of elements; a processor; a memory; and a communication interface. The processor is configured to determine whether each respective tier of elements satisfies each of a plurality of intrinsic observer capabilities, a plurality of intrinsic reactor capabilities, a plurality of first health checks received from an internal tier, and a plurality of second health checks received from an external tier. When any of the intrinsic observer capabilities and the intrinsic reactor capabilities are not satisfied, an extrinsic observer capability and/or an extrinsic reactor capability is used to compensate for the unsatisfied capability. When any of health checks discover a degradation of service, communication flows are routed so as to fully or partially avoid the affected tier of elements.
1. A method for mitigating a failure in a distributed cloud-based data center that includes at least two availability zones, each respective availability zone including a corresponding plurality of components, the method being implemented by at least one processor, the method comprising:
checking, by the at least one processor, each respective component from among a first plurality of components included in a first availability zone;
detecting, by the at least one processor, at least one partial failure that is associated with at least one component from among the first plurality of components;
routing, by the at least one processor, at least one communication flow so as to avoid the at least one component for which the at least one partial failure has been detected;
generating, by the at least one processor, a notification message that includes information that relates to the detected at least one partial failure; and
transmitting, by the at least one processor, the notification message to a predetermined destination,
wherein the routing comprises routing the at least one communication flow so as to avoid the first availability zone, and
wherein the method further comprises:
determining, based on a result of checking each respective component from among a second plurality of components included in a second availability zone, that all components in the second plurality of components are functioning normally,
wherein the routing further comprises routing the at least one communication flow so as to propagate via the second availability zone.
2. The method of claim 1 , wherein the checking comprises measuring, for each respective component, at least one health metric that indicates whether the respective component is functioning normally.
3. The method of claim 2 , wherein the checking further comprises:
generating a synthetic service request;
routing the synthetic service request so as to propagate via each of the first plurality of components;
receiving, from each respective component, a corresponding response to the synthetic service request; and
using each received response to measure each corresponding at least one health metric.
4. The method of claim 1 , further comprising:
receiving information indicating that an operational functionality has been restored for the at least one component for which the at least one partial failure has been detected; and
rerouting the at least one communication flow so as to propagate through the at least one component for which the operational functionality has been restored.
5. The method of claim 1 , further comprising updating a database with information that relates to a number of partial failures that are detected within a predetermined time interval.
6. The method of claim 5 , further comprising updating the database with information that relates to a number of components that are determined as functioning normally within the predetermined time interval.
7. The method of claim 1 , wherein the at least one partial failure includes at least one from among a hardware failure, a network disruption, a communication overload, a performance degradation, a random packet loss, an input/output glitch, a memory thrashing, a capacity pressure, and a non-fatal exception.
8. A computing apparatus for mitigating a failure in a distributed cloud-based data center that includes at least two availability zones, each respective availability zone including a corresponding plurality of components, the computing apparatus comprising:
a processor;
a memory; and
a communication interface coupled to each of the processor and the memory,
wherein the processor is configured to:
check each respective component from among a first plurality of components included in a first availability zone;
detect at least one partial failure that is associated with at least one component from among the first plurality of components;
route at least one communication flow so as to avoid the at least one component for which the at least one partial failure has been detected;
generate a notification message that includes information that relates to the detected at least one partial failure; and
transmit, via the communication interface, the notification message to a predetermined destination,
wherein the processor is further configured to route the at least one communication flow so as to avoid the first availability zone, and
wherein the processor is further configured to:
determine, based on a result of checking each respective component from among a second plurality of components included in a second availability zone, that all components in the second plurality of components are functioning normally; and
route the at least one communication flow so as to propagate via the second availability zone.
9. The computing apparatus of claim 8 , wherein the processor is further configured to measure, for each respective component, at least one health metric that indicates whether the respective component is functioning normally.
10. The computing apparatus of claim 9 , wherein the processor is further configured to:
generate a synthetic service request;
route the synthetic service request so as to propagate via each of the first plurality of components;
receive, from each respective component, a corresponding response to the synthetic service request; and
use each received response to measure each corresponding at least one health metric.
11. The computing apparatus of claim 8 , wherein the processor is further configured to:
receive information indicating that an operational functionality has been restored for the at least one component for which the at least one partial failure has been detected; and
reroute the at least one communication flow so as to propagate through the at least one component for which the operational functionality has been restored.
12. The computing apparatus of claim 8 , wherein the processor is further configured to update a database with information that relates to a number of partial failures that are detected within a predetermined time interval.
13. The computing apparatus of claim 12 , wherein the processor is further configured to update the database with information that relates to a number of components that are determined as functioning normally within the predetermined time interval.
14. The computing apparatus of claim 8 , wherein the at least one partial failure includes at least one from among a hardware failure, a network disruption, a communication overload, a performance degradation, a random packet loss, an input/output glitch, a memory thrashing, a capacity pressure, and a non-fatal exception.
15. An automatically self-healing multi-tier system for providing seamless resiliency, comprising:
a plurality of tiers of elements;
a processor;
a memory; and
a communication interface coupled to the processor, the memory, and each respective tier of elements from among the plurality of tiers of elements,
wherein the processor is configured to:
determine whether each respective tier of elements satisfies each of a plurality of intrinsic observer capabilities, a plurality of intrinsic reactor capabilities, a plurality of first health checks received from an internal tier, and a plurality of second health checks received from an external tier;
when at least one from among the plurality of intrinsic observer capabilities and the plurality of intrinsic reactor capabilities is not satisfied, use at least one from among a plurality of extrinsic observer capabilities and a plurality of extrinsic reactor capabilities to compensate for the unsatisfied capability; and
when at least one from among the plurality of first health checks and the plurality of second health checks is not satisfied, route at least one communication flow so as to avoid the tier of elements for which the corresponding health check is not satisfied.
16. The system of claim 15 , wherein the processor is further configured to measure, for each respective tier of elements, at least one health metric that indicates whether the respective tier of elements is well-behaved.