Method for detecting and resolving a partition condition in a cluster
View Patent ↗A method for detecting and resolving a partition condition in a cluster of computers in a networked environment is described. In one example, the method includes creating a scratch pad area and dividing the scratch pad into slots. Each slot is associated with a node within the cluster. A generation number and a list of known nodes are recorded in each slot when a change of membership occurs in the cluster. The slots are compared to ensure the generation number and the list of known nodes matches in each slot, and the partition condition is resolved by creating a list of surviving nodes and re-allocating appropriate resources to each of the surviving nodes.
1. A computer program product for detecting and resolving a partition condition in a cluster of computers in a networked environment, the computer program product stored on a non-transitory computer-readable medium and comprising:
computer-executable instructions for creating a scratch pad area accessible by the cluster of computers;
computer-executable instructions for dividing the scratch pad into a plurality of slots, each slot associated with one of a plurality of nodes within the cluster of computers, wherein each slot includes at least a heartbeat field indicating that cluster software is loaded on the node and a node state field indicating a current state of the node, wherein the current state identifies the node as being dead, alive, or preparing to shut down;
computer-executable instructions for recording in the plurality of slots, a generation number and a list of known nodes by each one of the plurality of nodes, wherein an identifier is written in the list for each node that is known to a writing node and wherein the generation number and the list of known nodes is recorded when a change of membership occurs in the cluster of computers;
computer-executable instructions for comparing each slot of the plurality of slots to ensure the generation number and the list of known nodes matches in each slot of the plurality of slots;
computer-executable instructions for resolving the partition condition by creating a list of surviving nodes and re-allocating appropriate resources to each of the surviving nodes,
computer-executable instructions requiring each node not on the list of surviving nodes to re-register with the cluster of computers; and
wherein the computer-executable instructions for comparing each slot include computer-executable instructions for finding a list with a master node to create the list of surviving nodes and shutting down each node not on the list with the master node.
2. The computer program product of claim 1 wherein the computer-executable instructions for creating the list of surviving nodes include computer-executable instructions for listing a first set of nodes determined by comparing each slot of the plurality of slots.
3. The computer program product of claim 2 wherein the computer-executable instructions for comparing each slot include computer-executable instructions for finding a list with a lowest node rank to create the list of surviving nodes and shutting down each node not on the list with the lowest node rank.
4. The computer program product of claim 2 wherein the computer-executable instructions for comparing each slot include computer-executable instructions for finding a list with a largest node to create the list of surviving nodes and shutting down each node not on the list with the largest node.
5. The computer program product of claim 2 wherein the computer-executable instructions for comparing each slot include computer-executable instructions for finding a list with a maximum number of nodes to create the list of surviving nodes and shutting down each node not on the list with the maximum number of nodes.
6. The computer program product of claim 5 further including computer-executable instructions for sending the list of surviving nodes to the each node on the list of surviving nodes along with a new generation number.
7. A method for detecting and resolving a partition condition in a cluster of computers in a networked environment, the method comprising:
creating a scratch pad area accessible by the cluster of computers;
dividing the scratch pad into a plurality of slots, each slot associated with one of a plurality of nodes within the cluster of computers;
recording in the plurality of slots, a generation number and a list of known nodes by each one of the plurality of nodes, wherein an identifier is written in the list for each node that is known to a writing node and wherein the generation number and the list of known nodes is recorded when a change of membership occurs in the cluster of computers;
comparing each slot of the plurality of slots to ensure the generation number and the list of known nodes matches in each slot of the plurality of slots;
creating a list of surviving nodes by listing a first set of nodes determined by comparing each slot of the plurality of slots, including finding a list with a master node to create the list of surviving nodes;
re-allocating appropriate resources to each of the surviving nodes; and
shutting down each node not on the list of surviving nodes with the master node by requiring each node not on the list of surviving nodes to write a special message in a respective slot for that node and then shut down immediately.
8. The method of claim 7 wherein the comparing each slot includes finding a list with a lowest node rank to create the list of surviving nodes and shutting down each node not on the list with the lowest node rank.
9. The method of claim 7 wherein the comparing each slot includes finding a list with a largest node to create the list of surviving nodes and shutting down each node not on the list with the largest node.
10. The method of claim 7 wherein the comparing each slot includes finding a list with a maximum number of nodes to create the list of surviving nodes and shutting down each node not on the list with the maximum number of nodes.
11. The method of claim 7 further including sending the list of surviving nodes to the each node on the list of surviving nodes along with a new generation number.
12. The method of claim 11 further including requiring each node not on the list of surviving nodes to re-register with the cluster of computers.
13. A method for detecting and resolving a partition condition in a cluster of computers in a networked environment, the method comprising:
creating a scratch pad area accessible by the cluster of computers;
dividing the scratch pad into a plurality of slots, each slot associated with one of a plurality of nodes within the cluster of computers, wherein each slot includes at least a heartbeat field indicating that cluster software is loaded on the node and a node state field indicating a current state of the node, wherein the current state identifies the node as being dead, alive, or preparing to shut down;
recording in the plurality of slots, a generation number and a list of known nodes by each one of the plurality of nodes, wherein an identifier is written in the list for each node that is known to a writing node and wherein the generation number and the list of known nodes is recorded when a change of membership occurs in the cluster of computers;
comparing each slot of the plurality of slots to ensure the generation number and the list of known nodes matches in each slot of the plurality of slots;
resolving the partition condition by creating a list of surviving nodes and re-allocating appropriate resources to each of the surviving nodes;
requiring each node not on the list of surviving nodes to re-register with the cluster of computers; and
instructing all nodes in the cluster to shut down if at least one non-surviving node fails to update its node state to indicate that it is not a surviving node.
14. A method for detecting and resolving a partition condition in a cluster of computers in a networked environment, the method comprising:
maintaining a scratch pad area accessible by the cluster of computers, wherein the scratch pad area is divided into a plurality of slots and each slot is associated with at least one of a plurality of nodes within the cluster of computers;
recording in the plurality of slots a generation number, a list of known nodes by each one of the plurality of nodes, and a node state indicating whether the node is a surviving node within the cluster, wherein the generation number and the list of known nodes is recorded when a change of membership occurs in the cluster of computers, and wherein the node state is updated at least when the node is not a surviving node;
comparing each slot of the plurality of slots to ensure the generation number and the list of known nodes matches in each slot of the plurality of slots;
resolving the partition condition by creating a list of surviving nodes, wherein nodes not on the list of surviving nodes update their node state to indicate that they are not a surviving node; and
instructing all nodes in the cluster to shut down if at least one non-surviving node fails to update its node state to indicate that it is not a surviving node.
15. The method of claim 14 further comprising publishing the list containing the surviving nodes only after all non-surviving nodes have updated their node state to indicate that they are not a surviving node.
16. The method of claim 7 , wherein the dividing the scratch pad includes dividing the scratch pad into the plurality of slots, wherein each slot includes at least a node state field indicating a current state of the node, wherein the current state identifies the node as being dead, alive, or preparing to shut down.
17. The method of claim 14 , wherein the maintaining a scratch pad area includes maintaining the scratch pad area accessible by the cluster of computers, wherein each slot includes at least a node state field indicating a current state of the node, wherein the current state identifies the node as being dead, alive, or preparing to shut down.