Hierarchically decisioned failover system

1 . A system, comprising:

a computing device comprising a processor and a memory; and

machine-readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least:

determine that an application instance among a plurality of application instances has failed to provide a health status report;

send a heartbeat request to the application instance for the health status report;

determine that the application instance failed to respond to the heartbeat request;

generate and save a separate health status report for the application instance in response to a determination that the application instance failed to respond to the heartbeat request, the separate health status report comprising a maximum weight;

calculate a respective instance health score for each of the plurality of application instances within an application service, wherein each respective instance health score is based at least in part on a respective plurality of health status reports for each of the plurality of application instances within a period of time, and wherein each of the respective plurality of health status reports for each of the plurality of application instances is based at least in part on a weight associated with a relative importance of a type of error;

calculate a service health score for the application service, wherein the application service comprises the plurality of application instances and the service health score is based at least in part on the respective instance health score for each of the plurality of application instances;

calculate a zone health score for an application ecosystem based at least in part on a plurality of service health scores, wherein the application ecosystem comprises a plurality of application services and each of the plurality of service health scores is associated with a respective one of the plurality of application services within the application ecosystem, and wherein the service health score for the respective one of the plurality of application services is based at least in part on a weight associated with a relative importance of the respective one of the plurality of application services;

initiate a failover of the application ecosystem from a first availability zone to a second availability zone based at least in part on the zone health score; and

migrate the application ecosystem from the first availability zone to the second availability zone in response to the failover.

2 . The system of claim 1 , wherein the machine-readable instructions further cause the computing device to at least:

determine that the zone health score exceeds a predefined threshold; and

initiate the failover in response to a determination that the zone health score exceeds the predefined threshold.

3 . The system of claim 1 , wherein the machine-readable instructions that cause the computing device to calculate the respective instance health score for each of the plurality of application instances, when executed, further cause the computing device to at least:

calculate a respective weighted average score for each of the plurality of application instances, wherein the respective weighted average score is based at least in part upon the respective plurality of health status reports for each of the plurality of application instances within the period of time, each of the respective plurality of health status reports representing a health event, a weight for the health event, and an availability of each application instance.

4 . The system of claim 3 , wherein the weight for the health event represents a severity of the health event.

5 . The system of claim 1 , wherein the machine-readable instructions that cause the computing device to calculate the service health score for the application service, when executed by the processor, further cause the computing device to at least calculate an average score from the respective instance health score for each of the plurality of application instances within the application service.

6 . The system of claim 1 , wherein each service health score for the respective one of the plurality of application services within the application ecosystem has a predefined weight and the machine-readable instructions that cause the computing device to calculate the zone health score further cause the computing device to at least calculate the zone health score as a weighted average of each service health score for each respective application service of the plurality of application services.

7 . The system of claim 1 , wherein the machine-readable instructions that, when executed by the processor, cause the computing device to determine that the application instance among the plurality of application instances has failed to provide the health status report further cause the computing device to at least:

predict an expected number of health status reports from the application instance among the plurality of application instances within the period of time;

determine a number of health status reports from the application instance among the plurality of application instances deviates from the expected number of health status reports; and

determine the application instance among the plurality of application instances failed to provide the expected number of health status reports.

8 . A method, comprising:

determining that an application instance among a plurality of application instances has failed to provide a health status report;

sending a heartbeat request to the application instance for the health status report;

determining that the application instance failed to respond to the heartbeat request;

generating and saving a separate health status report for the application instance in response to a determination that the application instance failed to respond to the heartbeat request, the separate health status report comprising a maximum weight;

calculating a respective instance health score for each of the plurality of application instances within an application service, wherein each respective instance health score is based at least in part on a respective plurality of health status reports for each of the plurality of application instances within a period of time, and wherein each of the respective plurality of reported health scores for each of the plurality of application instances is based at least in part on a weight associated with a relative importance of a type of error;

calculating a service health score for the application service, wherein the application service comprises the plurality of application instances and the service health score is based at least in part on the respective instance health score for each of the plurality of application instances;

calculating a zone health score for an application ecosystem based at least in part on a plurality of service health scores, wherein the application ecosystem comprises a plurality of application services and the service health score is one of a plurality of service health scores for a respective one of the plurality of application services within the application ecosystem, and wherein the service health score for the respective one of the plurality of application services is based at least in part on a weight associated with a relative importance of the respective one of the plurality of application services;

initiating a failover of the application ecosystem from a first availability zone to a second availability zone based at least in part on the zone health score; and

migrating the application ecosystem from the first availability zone to the second availability zone in response to the failover.

9 . The method of claim 8 , further comprising:

determining that the zone health score exceeds a predefined threshold; and

initiating the failover in response to a determination that the zone health score exceeds the predefined threshold.

10 . The method of claim 8 , wherein calculating the respective instance health score for each of the plurality of application instances further comprises calculating a respective weighted average score for each of the plurality of application instances, wherein the respective weighted average score is based at least in part upon a respective plurality of health status reports for each of the plurality of application instances within the period of time, each of the respective plurality of health status reports representing a health event, a weight for the health event, and an availability of each application instance.

11 . The method of claim 10 , wherein the weight for the health event represents a severity of the health event.

12 . The method of claim 8 , wherein calculating the service health score for the application service further comprises calculating an average score from the respective instance health score for each of the plurality of application instances within the application service.

13 . The method of claim 8 , wherein each service health score for the respective one of the plurality of application services within the application ecosystem has a predefined weight and calculating the zone health score further comprises calculating the zone health score as a weighted average of each service health score for each respective application service of the plurality of application services.

14 . The method of claim 8 , wherein determining that the application instance among the plurality of application instances has failed to provide the health status report further comprises:

predicting an expected number of health status reports from the application instance among the plurality of application instances within the period of time;

determining a number of health status reports from the application instance among the plurality of application instances deviates from the expected number of health status reports; and

determining the application instance among the plurality of application instances failed to provide the expected number of health status reports.

15 . A non-transitory computer-readable medium, comprising machine-readable instructions that, when executed by a processor of a computing device, cause the computing device to at least:

determine that an application instance among a plurality of application instances has failed to provide a health status report;

send a heartbeat request to the application instance for the health status report;

determine that the application instance failed to respond to the heartbeat request;

generate and save a separate health status report for the application instance in response to a determination that the application instance failed to respond to the heartbeat request, the separate health status report comprising a maximum weight;

calculate a respective instance health score for each of the plurality of application instances within an application service, wherein each respective instance health score is based at least in part on a respective plurality of health status reports for each of the plurality of application instances within a period of time, and wherein each of the respective plurality of health status reports for each of the plurality of application instances is based at least in part on a weight associated with a relative importance of a type of error;

calculate a service health score for the application service, wherein the application service comprises the plurality of application instances and the service health score is based at least in part on the respective instance health score for each of the plurality of application instances;

calculate a zone health score for an application ecosystem based at least in part on a plurality of service health scores, wherein the application ecosystem comprises a plurality of application services and the service health score is one of a plurality of service health scores for a respective one of the plurality of application services within the application ecosystem, and wherein the service health score for the respective one of the plurality of application services is based at least in part on a weight associated with a relative importance of the respective one of the plurality of application services;

initiate a failover of the application ecosystem from a first availability zone to a second availability zone based at least in part on the zone health score; and

migrate the application ecosystem from the first availability zone to the second availability zone in response to the failover.

16 . The non-transitory computer-readable medium of claim 15 , wherein the machine-readable instructions further cause the computing device to at least:

determine that the zone health score exceeds a predefined threshold; and

initiate the failover in response to a determination that the zone health score exceeds the predefined threshold.

17 . The non-transitory computer-readable medium of claim 15 , wherein the machine-readable instructions that cause the computing device to calculate the respective instance health score for each of the plurality of application instances, when executed, further cause the computing device to at least:

calculate a respective weighted average score for each of the plurality of application instances, wherein the respective weighted average score is based at least in part upon a respective plurality of health status reports for each of the plurality of application instances within the period of time, each of the respective plurality of health status reports representing a health event, a weight for the health event, and an availability of each application instance.

18 . The non-transitory computer-readable medium of claim 15 , wherein the machine-readable instructions that cause the computing device to calculate the service health score for the application service, when executed by the processor, further cause the computing device to at least calculate an average score from the respective instance health score for each of the plurality of application instances within the application service.

19 . The non-transitory computer-readable medium of claim 15 , wherein each service health score for the respective one of the plurality of application services within the application ecosystem has a predefined weight and the machine-readable instructions that cause the computing device to calculate the zone health score further cause the computing device to at least calculate the zone health score as a weighted average of each service health score for each respective application service of the plurality of application services.

20 . The non-transitory computer-readable medium of claim 15 , wherein the machine-readable instructions that, when executed by the processor, cause the computing device to determine that the application instance among the plurality of application instances has failed to provide the health status report further cause the computing device to at least:

predict an expected number of health status reports from the application instance among the plurality of application instances within the period of time;

determine a number of health status reports from the application instance among the plurality of application instances deviates from the expected number of health status reports; and

determine the application instance among the plurality of application instances failed to provide the expected number of health status reports.