Detecting and correcting node misconfiguration of information about the location of shared storage resources
A group of shared-storage resources are being maintained as replicas by the nodes of a cluster. Each node in the cluster maintains a set of pointers to the members of the group. As membership changes, the set of pointers should be changed accordingly. For various, reasons, the set of pointers of a node are not changed and may point to shared-storage resources that are no longer members of the group and/or not point to shared-storage resources that are members of the group. By recording a history of membership changes in the shared-storage resources, and comparing the recording to what a node thinks the shared resources should be, an incorrect set of pointers can be detected and corrective active undertaken.
1. A method for handling misconfigured pointers stored on a cluster that should point to
a shared-storage resource of the cluster, comprising:
changing membership of a set of one or more shared-storage resources to create versions of the set;
wherein the cluster includes a plurality of nodes that includes a first node;
for each version of the set, generating a membership stamp that:
is generated based on pointers to the one or more members of said each version, and
represents the membership of said each version of the set;
for each version of the set, storing in each member of said each version said each version's respective membership stamp;
storing one or more pointers on the first node, said one or more pointers pointing to a certain version of the set of one or more shared-storage resources;
generating a local membership stamp based on the one or more pointers, wherein said local membership reflects the membership of the certain version of the set of one or more shared-storage resource;
making a comparison of the local membership stamp to the membership stamps stored in at least a portion of the certain version of the set; and
making a determination of whether the one or more pointers point to all the members of the latest version of the set, said determination being based on one or more criteria, said one or more criteria being based on the comparison.
2. The method of claim 1 , wherein the set includes N replicated shared-storage resources, wherein N is greater than or equal to two.
3. The method of claim 2 , wherein the one or more criteria include:
changing membership of the set removes no more than X % members, and
the certain set includes a number of members that is greater than 100%-X % members.
4. The method of claim 1 , wherein the steps further include, in response to determining that one or more pointer do not point to all the members of the latest version of the set, said first node automatically changing the one or more pointers to point to the latest version of the set.
5. The method of claim 4 , wherein:
the steps further include transmitting messages to other nodes in the cluster requesting information about the membership of the latest version of the set;
receiving the information requested; and
changing the one or more pointers is based on the information received.
6. The method of claim 1 , wherein storing in each member of said each version said each version's respective membership stamp includes storing in a particular member the respective membership stamp along with a previously generated membership stamp of a previous version of the set.
7. The method of claim 1 , wherein each membership stamp generated for a version is generated by applying a hash function to a particular pointer to a shared-storage resource.
8. The method of claim 7 , wherein the particular pointer is a path string.
9. The method of claim 1 , wherein each member of each of version of the set is a cluster registry for the cluster.
10. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 1 .
11. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 2 .
12. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 3 .
13. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 4 .
14. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 5 .
15. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 6 .
16. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 7 .
17. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 8 .
18. A computer-readable medium carrying one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method recited in claim 9 .