Method for detecting and resolving a partition condition in a cluster
View Patent ↗A method and system to detect and resolve a partition condition in a cluster of computers in a networked environment is described. The method can include: creating a scratch pad area accessible by the cluster of computers; dividing the scratch pad into a plurality of slots; recording in the plurality of slots, a generation number and a list of known nodes by each one of the plurality of notes, wherein an identifier is written in the list for each node that is known to a writing node; comparing each slot of the plurality of slots to ensure the generation number and the list of known nodes matches in each slot of the plurality of slots; and resolving the partition condition by creating a list of surviving nodes and re-allocating appropriate resources to each of the surviving nodes.
1. A method for detecting and resolving a partition condition in a cluster of computers in a networked environment, the method comprising:
creating a scratch pad area accessible by the cluster of computers;
dividing the scratch pad into a plurality of slots, each slot associated with one of a plurality of nodes within the cluster of computers;
recording in the plurality of slots, a generation number and a list of known nodes by each one of the plurality of nodes, wherein an identifier is written in the list for each node that is known to a writing node and wherein the generation number and the list of known nodes is recorded when a change of membership occurs in the cluster of computers;
comparing each slot of the plurality of slots to ensure the generation number and the list of known nodes matches in each slot of the plurality of slots;
resolving the partition condition by creating a list of surviving nodes and re-allocating appropriate resources to each of the surviving nodes; and
ordering a first node not on the list of surviving nodes to halt execution by writing, by a second node on the list of surviving nodes, a termination message into the slot associated with the first node.
2. The method of claim 1 wherein the creating the list of surviving nodes includes listing a first set of nodes determined by comparing each slot of the plurality of slots.
3. The method of claim 2 wherein the comparing each slot includes finding a list with a master node to create the list of surviving nodes and shutting down each node not on the list with the master node.
4. The method of claim 2 wherein the comparing each slot includes finding a list with a lowest node rank to create the list of surviving nodes and shutting down each node not on the list with the lowest node rank.
5. The method of claim 2 wherein the comparing each slot includes finding a list with a largest node to create the list of surviving nodes and shutting down each node not on the list with the largest node.
6. The method of claim 2 wherein the comparing each slot includes finding a list with a maximum number of nodes to create the list of surviving nodes and shutting down each node not on the list with the maximum number of nodes.
7. The method of claim 2 further including shutting down each node not on the list of surviving nodes by requiring each node not on the list of surviving nodes to write a special message in a respective slot for that node and then shut down immediately.
8. The method of claim 7 further including sending the list of surviving nodes to the each node on the list of surviving nodes along with a new generation number.
9. The method of claim 7 further including requiring each node not on the list of surviving nodes to re-register with the cluster of computers.