Preventing data corruption due to pre-existing split brain
View Patent ↗Various methods, systems, and processes to prevent data corruption caused by a pre-existing split brain condition in a cluster are presented. In response to determining that a node is no longer part of a sub-cluster, another node in the sub-cluster is designated as a racer node. The racer node accesses a matrix, and the matrix includes information to determine whether a majority of coordination points in a cluster are accessible by nodes in the sub-cluster. Based on the accessing, a determination is made that the information indicates that the majority of coordination points are accessible by the nodes. The information is then broadcasted.
1. A computer-implemented method comprising:
in response to determining that a node is no longer part of a sub-cluster of a cluster, designating another node in the sub-cluster as a racer node;
the racer node accessing a matrix, wherein
the matrix comprises information to determine whether a majority of a plurality of coordination points in the cluster are accessible during a fence race by a plurality of nodes in the sub-cluster, wherein each of the coordination points can fence off one or more nodes in the cluster from accessing one or more storage devices, and
the plurality of coordination points comprise an odd number of coordination points;
based on the accessing, determining that the information indicates that the majority of the plurality of coordination points are accessible by the plurality of nodes; and
broadcasting the information.
2. The computer-implemented method of claim 1 , wherein
the node that is no longer part of the sub-cluster, previously had access to the majority of the plurality of coordination points.
3. The computer-implemented method of claim 1 , further comprising:
after determining that the node is no longer part of the sub-cluster, and prior to designating the another node as the racer node, determining that no node in the sub-cluster had access to the majority of the plurality of coordination points.
4. The computer-implemented method of claim 1 , wherein
broadcasting the information comprises updating the matrix with the information indicating that the majority of the plurality of coordination points are accessible by the plurality of nodes in the sub-cluster, wherein
the information is broadcasted to the cluster, and
the information also indicates that a fencing race performed by the racer node is successful.
5. The computer-implemented method of claim 4 , further comprising:
based on the information, inhibiting a partitioned node from accessing one or more coordination points of the majority of the plurality of coordination points accessible by the plurality of nodes in the sub-cluster, wherein
the inhibiting prevents a pre-existing split brain condition from occurring in the cluster.
6. The computer-implemented method of claim 1 , wherein
the matrix is a coordination point reachability matrix (CPRM), and
the CPRM is replicated across all nodes of the plurality of nodes in the cluster.
7. The computer-implemented method of claim 1 , wherein
the plurality of coordination points comprise
one or more coordinator disks, or
one or more coordination point servers.
8. A non-transitory computer readable storage medium storing program instructions executable for:
in response to determining that a node is no longer part of a sub-cluster of a cluster,
designating another node in the sub-cluster as a racer node;
accessing a matrix, wherein
the matrix comprises information to determine whether a majority of a plurality of coordination points in the cluster are accessible during a fence race by a plurality of nodes in the sub-cluster, wherein each of the coordination points can fence off one or more nodes in the cluster from accessing one or more storage devices, and
the plurality of coordination points comprise an odd number of coordination points;
based on the accessing, determining that the information indicates that the majority of the plurality of coordination points are accessible by the plurality of nodes; and
broadcasting the information.
9. The non-transitory computer readable storage medium of claim 8 , wherein the node that is no longer part of the sub-cluster, previously had access to the majority of the plurality of coordination points.
10. The non-transitory computer readable storage medium of claim 8 , wherein the program instructions are further executable for:
after determining that the node is no longer part of the sub-cluster, and prior to designating the another node as the racer node, determining that no node in the sub-cluster had access to the majority of the plurality of coordination points.
11. The non-transitory computer readable storage medium of claim 8 , wherein
broadcasting the information comprises updating the matrix with the information indicating that the majority of the plurality of coordination points are accessible by the plurality of nodes in the sub-cluster, wherein
the information is broadcasted to the cluster, and
the information also indicates that a fencing race performed by the racer node is successful.
12. The non-transitory computer readable storage medium of claim 11 , wherein the program instructions are further executable for:
based on the information, inhibiting a partitioned node from accessing one or more coordination points of the majority of the plurality of coordination points accessible by the plurality of nodes in the sub-cluster, wherein
the inhibiting prevents a pre-existing split brain condition from occurring in the cluster.
13. The non-transitory computer readable storage medium of claim 8 , wherein
the matrix is a coordination point reachability matrix (CPRM), and
the CPRM is replicated across all nodes of the plurality of nodes in the cluster.
14. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors for:
in response to determining that a node is no longer part of a sub-cluster of a cluster,
designating another node in the sub-cluster as a racer node;
the racer node accessing a matrix, wherein
the matrix comprises information to determine whether a majority of a plurality of coordination points in the cluster are accessible during a fence race by a plurality of nodes in the sub-cluster, wherein each of the coordination points can fence off one or more nodes in the cluster from accessing one or more storage devices, and
the plurality of coordination points comprise an odd number of coordination points;
based on the accessing, determining that the information indicates that the majority of the plurality of coordination points are accessible by the plurality of nodes; and
broadcasting the information.
15. The system of claim 14 , wherein
the node that is no longer part of the sub-cluster previously, had access to the majority of the plurality of coordination points.
16. The system of claim 14 wherein the program instructions are further executable for:
after determining that the node is no longer part of the sub-cluster, and prior to designating the another node as the racer node, determining that no node in the sub-cluster had access to the majority of the plurality of coordination points.
17. The system of claim 14 , wherein
broadcasting the information comprises updating the matrix with the information indicating that the majority of the plurality of coordination points are accessible by the plurality of nodes in the sub-cluster, wherein
the information is broadcasted to the cluster, and
the information also indicates that a fencing race performed by the racer node is successful.
18. The system of claim 17 wherein the program instructions are further executable for:
based on the information, inhibiting a partitioned node from accessing one or more coordination points of the majority of the plurality of coordination points accessible by the plurality of nodes in the sub-cluster, wherein
the inhibiting prevents a pre-existing split brain condition from occurring in the cluster.
19. The system of claim 14 , wherein
the matrix is a coordination point reachability matrix (CPRM), and
the CPRM is replicated across all nodes of the plurality of nodes in the cluster.
20. The system of claim 14 , wherein
the plurality of coordination points comprise
one or more coordinator disks, or
one or more coordination point servers.