Systems and methods for managing sub-clusters within dependent clustered computing systems subsequent to partition events
View Patent ↗A computer-implemented method for managing sub-clusters within dependent clustered computing systems subsequent to partition events may include (1) identifying a first computing cluster and a second computing cluster, (2) determining that the first computing cluster depends on the second computing cluster, (3) detecting a partition event that partitions the first computing cluster into multiple sub-clusters, (4) determining, in response to detecting the partition event, whether the second computing cluster is partitioned into multiple sub-clusters, and (5) making, based at least in part on determining whether the second computing cluster is partitioned into multiple sub-clusters, a fencing decision for the multiple sub-clusters of the first computing cluster. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for managing sub-clusters within dependent clustered computing systems subsequent to partition events, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
identifying a first computing cluster and a second computing cluster;
determining that the first computing cluster comprises a virtual machine that is hosted by the second computing cluster such that the first computing cluster depends on the second computing cluster;
detecting a partition event that partitions the first computing cluster into multiple sub-clusters that are unable to communicate with one another;
determining, in response to detecting the partition event, whether the second computing cluster is partitioned into multiple sub-clusters;
making, based at least in part on determining whether the second computing cluster is partitioned into multiple sub-clusters, a fencing decision for the multiple sub-clusters of the first computing cluster.
2. The computer-implemented method of claim 1 , wherein making the fencing decision for the multiple sub-clusters of the first computing cluster comprises receiving, at a node of the second computing cluster and from a node of a sub-cluster of the first computing cluster, a request for the fencing decision for the multiple sub-clusters of the first computing cluster.
3. The computer-implemented method of claim 2 , wherein making the fencing decision for the multiple sub-clusters of the first computing cluster further comprises making, at the node of the second computing cluster and based at least in part on determining that the second computing cluster is not partitioned into multiple sub-clusters, the fencing decision for the multiple sub-clusters of the first computing cluster.
4. The computer-implemented method of claim 2 , wherein making the fencing decision for the multiple sub-clusters of the first computing cluster further comprises:
sending, from the node of the second computing cluster and in response to determining that the second computing cluster is partitioned into multiple sub-clusters, a request to a central coordination point for a fencing decision for the multiple sub-clusters of the second computing cluster;
receiving, at the node of the second computing cluster, the fencing decision for the multiple sub-clusters of the second computing cluster;
making, at the node of the second computing cluster and based at least in part on the fencing decision for the multiple sub-clusters of the second computing cluster, the fencing decision for the multiple sub-clusters of the first computing cluster.
5. The computer-implemented method of claim 2 , wherein making the fencing decision for the multiple sub-clusters of the first computing cluster further comprises:
relaying, at the node of the second computing cluster and to a central coordination point, the request for the fencing decision for the multiple sub-clusters of the first computing cluster;
making, at the central coordination point, the fencing decision for the multiple sub-clusters of the first computing cluster.
6. The computer-implemented method of claim 5 , wherein the request for the fencing decision for the multiple sub-clusters of the first computing cluster is relayed to the central coordination point after waiting for a predetermined amount of time.
7. The computer-implemented method of claim 2 , wherein making the fencing decision for the multiple sub-clusters of the first computing cluster further comprises:
sending, at the node of the second computing cluster and in response to determining that the second computing cluster is partitioned into multiple sub-clusters, a combined request to a central coordination point for the fencing decision for the multiple sub-clusters of the first computing cluster and a fencing decision for the multiple sub-clusters of the second computing cluster;
making, at the central coordination point and in response to the combined request from the node of the second computing cluster, the fencing decision for the multiple sub-clusters of the first computing cluster.
8. The computer-implemented method of claim 2 , wherein:
the node of the second computing cluster comprises a hypervisor;
the node of the sub-cluster of the first computing cluster comprises the virtual machine;
determining that the virtual machine is hosted by the second computing cluster comprises determining that the virtual machine is running on the hypervisor;
the request for the fencing decision for the multiple sub-clusters of the first computing cluster is received over a direct communication channel between the hypervisor and the virtual machine.
9. The computer-implemented method of claim 1 , wherein:
the first computing cluster comprises a cluster of virtual machines that includes the virtual machine;
the second computing cluster comprises a cluster of hypervisors;
determining that the virtual machine is hosted by the second computing cluster comprises determining that the cluster of virtual machines runs on the cluster of hypervisors.
10. The computer-implemented method of claim 9 , wherein
determining that the cluster of virtual machines runs on the cluster of hypervisors comprises tracking the node of the cluster of hypervisors on which each node within the cluster of virtual machines runs.
11. A system for managing sub-clusters within dependent clustered computing systems subsequent to partition events, the system comprising:
an identifying module, stored in memory, that identifies a first computing cluster and a second computing cluster;
a dependency-determining module, stored in memory, that determines that the first computing cluster comprises a virtual machine that is hosted by the second computing cluster such that the first computing cluster depends on the second computing cluster;
a partition-detecting module, stored in memory, that:
detects a partition event that partitions the first computing cluster into multiple sub-clusters that are unable to communicate with one another;
determines, in response to detecting the partition event, whether the second computing cluster is partitioned into multiple sub-clusters;
a fencing module, stored in memory, that makes, based at least in part on determining whether the second computing cluster is partitioned into multiple sub-clusters, a fencing decision for the multiple sub-clusters of the first computing cluster;
at least one processor configured to execute the identifying module, the dependency-determining module, the partition-detecting module, and the fencing module.
12. The system of claim 11 , wherein the fencing module makes the fencing decision for the multiple sub-clusters of the first computing cluster by receiving, at a node of the second computing cluster and from a node of a sub-cluster of the first computing cluster, a request for the fencing decision for the multiple sub-clusters of the first computing cluster.
13. The system of claim 12 , wherein the fencing module makes the fencing decision for the multiple sub-clusters of the first computing cluster further by making, at the node of the second computing cluster and based at least in part on determining that the second computing cluster is not partitioned into multiple sub-clusters, the fencing decision for the multiple sub-clusters of the first computing cluster.
14. The system of claim 12 , wherein the fencing module makes the fencing decision for the multiple sub-clusters of the first computing cluster further by:
sending, from the node of the second computing cluster and in response to determining that the second computing cluster is partitioned into multiple sub-clusters, a request to a central coordination point for a fencing decision for the multiple sub-clusters of the second computing cluster;
receiving, at the node of the second computing cluster, the fencing decision for the multiple sub-clusters of the second computing cluster;
making, at the node of the second computing cluster and based at least in part on the fencing decision for the multiple sub-clusters of the second computing cluster, the fencing decision for the multiple sub-clusters of the first computing cluster.
15. The system of claim 12 , wherein the fencing module makes the fencing decision for the multiple sub-clusters of the first computing cluster further by:
relaying, at the node of the second computing cluster and to a central coordination point, the request for the fencing decision for the multiple sub-clusters of the first computing cluster;
making, at the central coordination point, the fencing decision for the multiple sub-clusters of the first computing cluster.
16. The system of claim 15 , wherein the fencing module waits for a predetermined amount of time before relaying the request for the fencing decision for the multiple sub-clusters of the first computing cluster.
17. The system of claim 12 , wherein the fencing module makes the fencing decision for the multiple sub-clusters of the first computing cluster further by:
sending, at the node of the second computing cluster and in response to determining that the second computing cluster is partitioned into multiple sub-clusters, a combined request to a central coordination point for the fencing decision for the multiple sub-clusters of the first computing cluster and a fencing decision for the multiple sub-clusters of the second computing cluster;
making, at the central coordination point and in response to the combined request from the node of the second computing cluster, the fencing decision for the multiple sub-clusters of the first computing cluster.
18. The system of claim 12 , wherein:
the node of the second computing cluster comprises a hypervisor;
the node of the sub-cluster of the first computing cluster comprises the virtual machine;
the dependency-determining module determines that the virtual machine is hosted by the second computing cluster by determining that the virtual machine is running on the hypervisor;
the request for the fencing decision for the multiple sub-clusters of the first computing cluster is received over a direct communication channel between the hypervisor and the virtual machine.
19. The system of claim 11 , wherein:
the first computing cluster comprises a cluster of virtual machines that includes the virtual machine;
the second computing cluster comprises a cluster of hypervisors;
the dependency-determining module determines that the virtual machine is hosted by the second computing cluster by determining that the cluster of virtual machines runs on the cluster of hypervisors.
20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
identify a first computing cluster and a second computing cluster;
determine that the first computing cluster comprises a virtual machine that is hosted by the second computing cluster such that the first computing cluster depends on the second computing cluster;
detect a partition event that partitions the first computing cluster into multiple sub-clusters that are unable to communicate with one another;
determine, in response to detecting the partition event, whether the second computing cluster is partitioned into multiple sub-clusters;
make, based at least in part on determining whether the second computing cluster is partitioned into multiple sub-clusters, a fencing decision for the multiple sub-clusters of the first computing cluster.