Failover for cloud-native applications
A system and method for failover for cloud-native applications, such as cloud-native video-conferencing services, that includes a software agent that monitors the health of services executing on servers. An agent monitors predefined metrics of a service and sends alerts to a plan-execution system when a monitored metric fails to satisfy a predefined condition. The plan-execution system applies predefined rules to received alerts to determine whether a service is failing. The plan-execution system instructs a command center to redirect client requests for a failing service to a failover service.
1 . A method, comprising:
monitoring, by a software agent executing on a first server in a distributed client-server platform, one or more metrics of a first service executing on the first server, wherein requests for the first service are forwarded to the first server by a command center;
ascertaining, by a plan-execution system, alerts related to the one or more metrics and determining whether the alerts satisfy a predefined rule; and
in response to the alerts satisfying the predefined rule, selecting, by the plan-execution system, a second server that is available and instructing the command center to redirect requests for the first service to a second service executing on the second server.
2 . The method of claim 1 , wherein the one or more metrics corresponds to:
a rate of successful or unsuccessful responses to requests received by the first service.
3 . The method of claim 1 , wherein the one or more metrics corresponds to:
an average response time of responses to requests received by the first service.
4 . The method of claim 1 , wherein the one or more metrics corresponds to:
receiving heartbeat messages from the first service.
5 . The method of claim 1 , wherein ascertaining an individual one of the alerts comprises:
receiving a message from the software agent indicating that a monitored metric of the one or more metrics failed to satisfy a predefined threshold.
6 . The method of claim 1 , wherein the predefined rule corresponds to:
a Boolean combination of the one or more alerts.
7 . The method of claim 1 , wherein:
an individual one of the alerts is discarded after a predefined timeout.
8 . The method of claim 1 , further comprising:
discarding individual ones of the alerts that are essential to the satisfying of the predefined rule upon instructing the command center to redirect the requests for the first service to the second service.
9 . The method of claim 1 , further comprising:
determining, by the plan-execution system, the second server that is available by sending a message to the second server requesting a response to the message from the second server.
10 . The method of claim 1 , further comprising:
generating test requests for the first service from simulated clients;
sending the test requests to the command center; and
determining a quantity of the test requests that were redirected to the second service.
11 . The method of claim 1 , wherein:
the distributed client-server platform comprises a video-conferencing platform.
12 . A non-transitory computer-readable medium storing instructions operable to cause one or more processors to perform operations comprising:
monitoring, by a software agent executing on a first server in a distributed client-server platform, one or more metrics of a first service executing on the first server, wherein requests for the first service are forwarded to the first server by a command center;
ascertaining, by a plan-execution system, alerts related to the one or more metrics and determining whether the alerts satisfy a predefined rule; and
in response to the alerts satisfying the predefined rule, selecting, by the plan-execution system, a second server that is available and instructing the command center to redirect requests for the first service to a second service executing on the second server.
13 . The medium of claim 12 , wherein the operations further comprise:
generating test requests for the first service from simulated clients;
sending the test requests to the command center; and
determining a quantity of the test requests that were redirected to the second service.
14 . The medium of claim 12 , wherein the one or more metrics correspond to at least one of:
a rate of successful or unsuccessful responses to requests received by the first service;
an average response time of responses to requests received by the first service; or
receiving heartbeat messages from the first service.
15 . The medium of claim 12 , wherein the instructing of the command center to redirect requests for the first service comprises:
instructing the command center to redirect fewer than all of the requests to the second service.
16 . A system, comprising:
one or more memories; and
one or more processors configured to execute instructions stored in the one or more memories to:
monitor, by a software agent executing on a first server in a distributed client-server platform, one or more metrics of a first service executing on the first server, wherein requests for the first service are forwarded to the first server by a command center;
ascertain, by a plan-execution system, alerts related to the one or more metrics and determine whether the alerts satisfy a predefined rule; and
in response to the alerts satisfying the predefined rule, select, by the plan-execution system, a second server that is available and instruct the command center to redirect requests for the first service to a second service executing on the second server.
17 . The system of claim 16 , wherein the instructions include instructions to:
generate test requests for the first service from simulated clients;
send the test requests to the command center; and
determine a quantity of the test requests that were redirected to the second service.
18 . The system of claim 16 , wherein the instructions include instructions to:
determine whether the alerts satisfy a predefined rule by determining whether each of a predefined quantity of the alerts ascertained within a predefined duration concerns a same metric that indicates a same error condition.
19 . The system of claim 16 , wherein the instructions include instructions to:
determine whether the alerts satisfy a predefined rule by determining whether each of a first predefined quantity of the alerts ascertained within a predefined duration concerns a same first metric that indicates a same first error condition and each of a second predefined quantity of the alerts ascertained within a second predefined duration concerns a same second metric that indicates a same second error condition.
20 . The system of claim 16 , wherein the instructions include instructions to:
determine, by the plan-execution system, the second server that is available by sending a message to the second server requesting a status of the second server.