System and method to prevent data corruption due to split brain in shared data clusters
A system and to prevent data corruption due to split brain in shared data clusters includes two or more nodes of a cluster, a shared storage device, and an update manager. The update manager may be configured to maintain a local persistent store corresponding to each node of the cluster. On receiving an update request directed to the shared storage device from a first node, the update manager may be configured to redirect the update to the local persistent store corresponding to the first node. The update manager may be further configured to verify a cluster membership status of the first node, and to transfer the contents of the update from the local persistent store to the shared storage device if the cluster membership verification succeeds.
1. A system, comprising:
a storage network;
a plurality of nodes coupled to the storage network, wherein the plurality of nodes includes at least two nodes that form a cluster, wherein the plurality of nodes includes a first node, and wherein the first node includes a first computer-accessible storage medium having stored thereon a first set of program instructions;
a plurality of local persistent stores, each of which is coupled to the storage network;
a shared storage device coupled to the storage network; and
a computer system coupled to the storage network, wherein the computer system includes a second computer-accessible storage medium having stored thereon a second set of program instructions;
wherein the first set of program instructions is executable by the first node to:
for an update directed to the shared storage device by the first node, redirect the update to a first of the plurality of local persistent stores, wherein the first local persistent store corresponds to the first node; and
in response to a successful verification of membership status of the first node in the cluster, cause contents of the update to be transferred from the first local persistent store to the shared storage device;
wherein the second set of program instructions is executable by the computer system to perform, at least in part, the verification of membership status of the first node in the cluster.
2. The system as recited in claim 1 , wherein the first set of program instructions is executable by the first node to:
transmit an update completion indication after contents of the update have been written to the local persistent store; and
transfer contents of the update to the shared storage device asynchronously with respect to a transmission of the update completion indication.
3. The system as recited in claim 1 , wherein the local persistent store is a local update log.
4. The system as recited in claim 3 , wherein the update includes one or more modified data blocks, and wherein, in redirecting the update to the local update log, the first set of program instructions is executable by the first node to cause a log entry corresponding to the one or more modified data blocks to be added to the local update log.
5. The system as recited in claim 3 , wherein an index to the one or more modified blocks is included in the log entry.
6. The system as recited in claim 1 , wherein the update includes one or more modified data blocks, and wherein, in redirecting the update, the first set of program instructions is executable to cause an entry to be inserted into the local persistent store, wherein the inserted entry includes the one or more modified blocks and cluster configuration information available at the first node.
7. The system as recited in claim 6 , wherein the cluster configuration information includes at least one of: a cluster generation number and an identification of current cluster member nodes;
and wherein the second set of program instructions is executable to use the cluster configuration information to perform the verification of the membership status of the first node.
8. The system as recited in claim 1 , wherein the first set of program instructions is executable to cause one or more cluster configuration entries to be inserted into the local persistent store, wherein each cluster configuration entry of the one or more cluster configuration entries includes cluster configuration information available at the first node, and wherein the second set of program instructions is executable to use at least one cluster configuration entry of the one or more cluster configuration entries to perform the verification of the membership status of the first node.
9. The system as recited in claim 8 , wherein the first set of program instructions is executable to cause a new cluster configuration entry to be inserted into the local persistent store in response to a change to a configuration of the cluster being detected at the first node.
10. The system as recited in claim 1 , further comprising a third set of program instructions executable to implement a recovery manager, wherein after a failure, the third set of program instructions is executable to:
make a determination as to whether a particular update at the local persistent store corresponding to a particular node of the cluster was performed while the particular node was in an invalid partition of the cluster; and
in response to the determination being positive, exclude the particular update from a set of updates, wherein the third set of program instructions is executable to cause, at least in part, contents of the set of updates, excluding the particular update, to be copied to the shared storage device in response to the particular node rejoining a valid partition of the cluster.
11. The system as recited in claim 1 , wherein the second set of program instructions is executable to determine whether the first node belongs to a quorum in accordance with a configurable quorum detection policy; and
wherein the membership status of the first node in the cluster is dependent on whether the first node belongs to the quorum.
12. The system as recited in claim 11 , wherein the quorum detection policy identifies a particular cluster partition as a quorum if the particular cluster partition includes a specified number of nodes.
13. The system as recited in claim 1 , wherein the local persistent store is a local database system.
14. The system as recited in claim 1 , wherein each node of the two or more nodes in the cluster corresponds to a different instance of a clustered application.
15. A method, comprising:
for an update directed to a shared storage device by a first node of a plurality of nodes within a storage network, wherein two or more of the plurality of nodes form a cluster, and wherein the shared storage device is coupled to the storage network:
a computer system executing stored program instructions to cause the computer system to perform operations including:
redirecting the update to a local persistent store corresponding to the first node, wherein the computer system is coupled to the shared storage device through the storage network, and wherein the local persistent store is one of a plurality of local persistent stores; and
in response to the computer system receiving an indication of a successful verification of membership status of the first node in the cluster, causing contents of the update from the local persistent store to be transferred to the shared storage device;
wherein the indication of a successful verification of membership status of the first node in the cluster is received from another computer system coupled to the storage network.
16. The method as recited in claim 15 , further comprising:
the computer system transmitting an update completion indication after contents of the update have been written to the local persistent store; and
wherein the transfer of contents of the update to the shared storage device is asynchronous with respect to the transmitting of the update completion indication.
17. The method as recited in claim 15 , wherein the update includes one or more modified data blocks, and wherein redirecting the update to the local persistent store includes adding a log entry corresponding to the one or more modified data blocks to the local persistent store.
18. The method as recited in claim 15 , further comprising the computer system:
causing cluster configuration relating to the first node to be inserted into the local persistent store; and
wherein the verification of membership status of the first node uses the cluster configuration information.
19. The method as recited in claim 15 , wherein, after a change to a configuration of the cluster, the verification of membership status of the first node includes determining whether the first node belongs to a quorum in accordance with a configurable quorum detection policy.
20. A computer-accessible storage medium comprising program instructions, wherein the instructions are executable by a computer system to cause the computer system to:
for an update directed to a shared storage device by a first node in a plurality of nodes within a storage network, wherein two or more of the plurality of nodes form a cluster:
redirect the update to a first local persistent store corresponding to the first node, wherein the first local persistent store is one of a plurality of local persistent stores; and
in response to the first node receiving an indication of a successful verification of membership status of the first node in the cluster, cause contents of the update to be transferred from the first local persistent store to the shared storage device;
wherein the indication of a successful verification of membership status of the first node in the cluster is received from another computer system connected to the storage network; and
wherein the shared storage device is connected to the storage network, and wherein the computer system is connected to the storage network.
21. The computer-accessible storage medium recited in claim 20 , wherein the instructions are further executable by the computer system to:
transmit an update completion indication after contents of the update have been written to the local persistent store; and
transfer contents of the update to the shared storage device asynchronously with respect to a transmission of the update completion indication.
22. The computer-accessible storage medium recited in claim 20 , wherein the update includes one or more modified data blocks, and wherein in redirecting the update to the local persistent store, the instructions are further executable by the computer system to cause a log entry corresponding to the one or more modified data blocks to be added to the local persistent store.
23. The computer-accessible storage medium recited in claim 20 , wherein the instructions are further executable by the computer system to:
cause cluster configuration available at the first node to be inserted into the local persistent store; and
wherein the verification of membership status of the first node in the cluster is based, at least in part, on the cluster configuration information.
24. The system of claim 1 , wherein the computer system is a second one of the plurality of nodes.
25. The system of claim 10 , wherein the third set of program instructions are stored on the second computer-accessible storage medium of the computer system coupled to the storage network.
26. The method as recited in claim 15 , wherein the computer system includes the first node.
27. The computer-accessible storage medium recited in claim 20 , wherein the computer system includes the first node.