Dynamic heartbeating mechanism
System and method for providing failover protection for a cluster of host computers operates in one of two modes. When the number of host computers in the cluster is equal to or less than a threshold number, the host computers in the cluster operate in an all-active heartbeating mode in which each of the host computers is enabled to function as a master node. When the number of host computers in the cluster is greater than the threshold number, the host computers in the cluster operate in a distributed heartbeating mode in which some of the host computers function as master nodes and at least one of the host computers functions as a slave node.
1. A method for providing failover protection for a cluster of host computers, the method comprising:
when the number of host computers in the cluster is equal to or less than a threshold number, operating the host computers in the cluster in an all-active heartbeating mode in which each of the host computers functions as a master node, including controlling a failover agent in each of the host computers to operate as a master so that each of the host computers functions as a master node; and
when the number of host computers in the cluster is greater than the threshold number, operating the host computers in the cluster in a distributed heartbeating mode in which at least one of the host computers functions as a master node and at least one of the host computers functions as a slave node, including controlling the failover agent in each of the host computers to operate as one of a master and a slave so that at least one of the host computers functions as a master node and at least one of the host computers functions as a slave node.
2. The method of claim 1 , wherein operating the host computers in the cluster in the all-active heartbeating mode or in the distributed heartbeating mode includes periodically sending a heartbeat message from each of the master nodes in the cluster to other master nodes in the cluster and sending a heartbeat response message from each of the master nodes that received the heartbeat message.
3. The method of claim 2 , wherein the heartbeat message includes updated local inventory information of a master mode sending the heartbeat message.
4. The method of claim 2 , wherein operating the host computers in the cluster in the all-active heartbeating mode or in the distributed heartbeating mode further includes performing at least one of a ping check and a storage heartbeat check on a particular master node in the cluster that did not send a heartbeat message to determine whether the particular master node has failed.
5. The method of claim 4 , wherein operating the host computers in the cluster in the all-active heartbeating mode or in the distributed heartbeating mode further includes, after the particular master node has been determined to have failed, competing to acquire a storage lock of the particular master node by other healthy master nodes in the cluster and initiating a failover operation on the particular master node by a healthy master node in the cluster that has acquired the storage lock.
6. The method of claim 5 , wherein operating the host computers in the cluster in the distributed heartbeating mode further includes periodically sending a slave heartbeat message from each slave node in the cluster to an associated master node in the cluster and sending a slave heartbeat response message from the associated master node back to each slave node.
7. The method of claim 6 , wherein operating the host computers in the cluster in the distributed heartbeating mode further includes performing at least one of a ping check and a storage heartbeat check on a particular slave node in the cluster that did not send a heartbeat message to determine whether the particular slave node has failed.
8. The method of claim 7 , wherein operating the host computers in the cluster in the distributed heartbeating mode further includes, after the particular slave node has been determined to have failed, initiating a failover operation on the particular slave node by the associated master node.
9. A non-transitory computer-readable storage medium containing program instructions for a method for providing failover protection for a cluster of host computers, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:
when the number of host computers in the cluster is equal to or less than a threshold number, operating the host computers in the cluster in an all-active heartbeating mode in which each of the host computers functions as a master node, including controlling a failover agent in each of the host computers to operate as a master so that each of the host computers functions as a master node; and
when the number of host computers in the cluster is greater than the threshold number, operating the host computers in the cluster in a distributed heartbeating mode in which at least one of the host computers functions as a master node and at least one of the host computers functions as a slave node, including controlling the failover agent in each of the host computers to operate as one of a master and a slave so that at least one of the host computers functions as a master node and at least one of the host computers functions as a slave node.
10. The computer-readable storage medium of claim 9 , wherein operating the host computers in the cluster in the all-active heartbeating mode or in the distributed heartbeating mode includes periodically sending a heartbeat message from each of the master nodes in the cluster to other master nodes in the cluster and sending a heartbeat response message from each of the master nodes that received the heartbeat message.
11. The computer-readable storage medium of claim 10 , wherein the heartbeat message includes updated local inventory information of a master mode sending the heartbeat message.
12. The computer-readable storage medium of claim 10 , wherein operating the host computers in the cluster in the all-active heartbeating mode or in the distributed heartbeating mode further includes performing at least one of a ping check and a storage heartbeat check on a particular master node in the cluster that did not send a heartbeat message to determine whether the particular master node has failed.
13. The computer-readable storage medium of claim 12 , wherein operating the host computers in the cluster in the all-active heartbeating mode or in the distributed heartbeating mode further includes, after the particular master node has been determined to have failed, competing to acquire a storage lock of the particular master node by other healthy master nodes in the cluster and initiating a failover operation on the particular master node by a healthy master node in the cluster that has acquired the storage lock.
14. The computer-readable storage medium of claim 13 , wherein operating the host computers in the cluster in the distributed heartbeating mode further includes periodically sending a slave heartbeat message from each slave node in the cluster to an associated master node in the cluster and sending a slave heartbeat response message from the associated master node back to each slave node.
15. The computer-readable storage medium of claim 14 , wherein operating the host computers in the cluster in the distributed heartbeating mode further includes performing at least one of a ping check and a storage heartbeat check on a particular slave node in the cluster that did not send a heartbeat message to determine whether the particular slave node has failed.
16. The computer-readable storage medium of claim 15 , wherein operating the host computers in the cluster in the distributed heartbeating mode further includes, after the particular slave node has been determined to have failed, initiating a failover operation on the particular slave node by the associated master node.
17. A distributed computer system comprising:
a cluster of host computers; and
a management server operatively connected to the host computers, the management server comprising:
memory; and
a processor configured to:
when the number of host computers in the cluster is equal to or less than a threshold number, manage the host computers in the cluster to operate in an all-active heartbeating mode in which each of the host computers functions as a master node, including controlling a failover agent in each of the host computers to operate as a master so that each of the host computers functions as a master node; and
when the number of host computers in the cluster is greater than the threshold number, manage the host computers in the cluster to operate in a distributed heartbeating mode in which at least one of the host computers functions as a master node and at least one of the host computers functions as a slave node, including controlling the failover agent in each of the host computers to operate as one of a master and a slave so that at least one of the host computers functions as a master node and at least one of the host computers functions as a slave node.
18. The distributed computer system of claim 17 , wherein the processor is configured to manage the host computers in the clusters so that each of the master nodes in the cluster periodically sends a heartbeat message to other master nodes in the cluster and each of the master nodes that received the heartbeat message sends a heartbeat response message.
19. The distributed computer system of claim 17 , wherein the heartbeat message includes updated local inventory information of a master mode sending the heartbeat message.
20. The distributed computer system of claim 19 , wherein the processor is configured to manage the host computers in the clusters so that healthy master nodes in the cluster perform at least one of a ping check and a storage heartbeat check on a particular master node in the cluster that did not send a heartbeat message to determine whether the particular master node has failed.
21. The distributed computer system of claim 20 , wherein the processor is configured to manage the host computers in the clusters so that, after the particular master node has been determined to have failed, other healthy master nodes in the cluster compete to acquire a storage lock of the particular master node and a healthy master node in the cluster that has acquired the storage lock initiates a failover operation on the particular master node.
22. The distributed computer system of claim 21 , wherein the processor is configured to manage the host computers in the clusters so that each slave node in the cluster periodically sends a slave heartbeat message to an associated master node in the cluster and the associated master nodes sends a slave heartbeat response message back to each slave node.
23. The distributed computer system of claim 22 , wherein the processor is configured to manage the host computers in the clusters so that the associated master node performs at least one of a ping check and a storage heartbeat check on a particular slave node in the cluster that did not send a heartbeat message to determine whether the particular slave node has failed.
24. The distributed computer system of claim 23 , wherein the processor is configured to manage the host computers in the clusters so that, after the particular slave node has been determined to have failed, the associated master node initiates a failover operation on the particular slave node.