Data storage cluster with witness node for selecting surviving storage node after replication failure
A clustered pair of data storage nodes employs a time-to-live (TTL) mechanism by which a preferred node communicates permission for continued operation to a non-preferred node. During non-errored TTL operation, host I/O requests to a data storage object are serviced, with write-type requests being replicated to the other node. Upon a failure as indicated by errored TTL operation or failure of replication, a polarization operation selects a surviving node to transition to single-node access to the data storage object. The polarization process includes: (1) each node contacting a witness node to request survivor status, (2) the witness node granting survivor status to the first node requesting it and denying survivor status to a later-requesting node, (3) at the node granted survivor status, continuing to service the host I/O requests without replication, and (4) at the other node based on being denied survivor status, discontinuing servicing of the host I/O requests.
1. A method of operating a clustered pair of data storage nodes, comprising:
employing a time-to-live (TTL) mechanism by which a preferred node of the clustered pair regularly communicates permission for continued operation to a non-preferred node of the clustered pair in response to regular TTL requests therefrom;
at both the preferred node and the non-preferred node during non-errored operation of the TTL mechanism, servicing host I/O requests to a data storage object and, for write-type requests, replicating the write-type requests to the other node of the clustered pair; and
detecting a failure as indicated by errored operation of the TTL mechanism or by failure of replicating the write-type requests, and performing a polarization operation to select a surviving node to transition to single-node access to the data storage object, the polarization process including: (1) at each of the nodes, contacting a third witness node to request survivor status, (2) at the witness node, granting survivor status to the first node requesting it and denying survivor status to a later-requesting node, (3) at the node granted survivor status, continuing to service host I/O requests to the data storage object without replicating write-type requests to the other node, and (4) at the other node, discontinuing servicing of host I/O requests to the data storage object.
2. The method according to claim 1 , wherein the TTL mechanism includes a TTL timer at the non-preferred node which is refreshed by each grant of TTL permission from the preferred node, and a corresponding tracking timer at the preferred node which tracks operation of the TTL timer, and wherein the non-errored operation includes regular refreshing of the TTL timer by each grant of TTL permission before the TTL timer expires, and wherein the errored operation includes expiration of the TTL timer.
3. The method according to claim 2 , wherein the errored operation results from an absence of grant of TTL permission from the preferred node notwithstanding a corresponding request of TTL from the non-preferred node that would normally result in a grant of TTL permission being sent.
4. The method according to claim 3 , wherein the failure condition is such that the preferred node is unable to send the grant of TTL permission.
5. The method according to claim 3 , wherein the failure condition includes failure of replicating a write request from the preferred node to the non-preferred node, and the absence of the grant of TTL permission is the intentional withholding of the grant by the preferred node in order to initiate the polarization operation.
6. The method according to claim 2 , wherein the errored operation results from an absence of grant of TTL permission from the preferred node due to absence of a preceding request of TTL from the non-preferred node that would normally result in a grant of TTL permission being sent.
7. The method according to claim 6 , wherein the failure condition is such that the non-preferred node is unable to send the request of TTL.
8. The method according to claim 6 , wherein the failure condition includes failure of replicating a write request from the non-preferred node to the preferred node, and the absence of the TTL request is the intentional withholding of the TTL request by the non-preferred node in order to initiate the polarization operation.
9. The method according to claim 1 , wherein the TTL mechanism includes use of an additional delay by the non-preferred node to provide a chance to the preferred node to be the first to contact the witness node and thereby obtain survivor status.
10. The method according to claim 1 , wherein the polarization operation at each node further includes queueing new host I/O requests, and (1) at the node granted survivor status, processing the queued host I/O requests as part of continuing to service host I/O requests, and (2) at the node denied survivor status, rejecting the queued host I/O requests.
11. The method according to claim 1 , wherein the node granted survivor status is the preferred node, and wherein the other node is the non-preferred node and discontinues servicing of host I/O requests based on being denied survivor status by the witness node.
12. The method according to claim 1 , wherein the node granted survivor status is the preferred node, and wherein the other node is the non-preferred node and discontinues servicing of host I/O requests based on expiration of its operating permission as indicated by the TTL mechanism.
13. The method according to claim 1 , wherein the node granted survivor status is the non-preferred node, and wherein the other node is the preferred node and discontinues servicing of host I/O requests based on being denied survivor status by the witness node.
14. A data storage node usable in a clustered pair of data storage nodes including a preferred node and a non-preferred node, the data storage node comprising a storage processor and physical storage, the storage processor storing computer program instructions executable by the storage processor to cause the data storage node to operate by:
employing a time-to-live (TTL) mechanism by which the preferred node regularly communicates permission for continued operation to the non-preferred node of the clustered pair in response to regular TTL requests therefrom;
during non-errored operation of the TTL mechanism, servicing host I/O requests to a data storage object and, for write-type requests, replicating the write-type requests to the other node of the clustered pair; and
detecting a failure as indicated by errored operation of the TTL mechanism or by failure of replicating the write-type requests, and performing a polarization operation to select a surviving node to transition to single-node access to the data storage object, the polarization process including: (1) contacting a third witness node to request survivor status, the witness node being configured and operable to grant survivor status to the first node requesting it and deny survivor status to a later-requesting node, (2) in response to being granted survivor status, continuing to service host I/O requests to the data storage object without replicating write-type requests to the other node, and (3) in response to being denied survivor status, discontinuing servicing of host I/O requests to the data storage object.
15. The data storage node according to claim 14 , configured for operation as the preferred node.
16. The data storage node according to claim 14 , configured for operation as the non-preferred node.