Data storage cluster with quorum service protection
In a protective quorum service, during an initial period of normal operation in which a clustered pair of data storage nodes provide host I/O access to a data storage object and replicate write-type requests to each other, the nodes are first registered to the quorum service. Subsequently, based on the registration and in response to a first auto promote request from a first-requesting node, a success response is returned and the service enters an auto promoted condition, the success response indicating that the first-requesting node is to continue providing the host I/O access to the data storage object without write replication. In response to receiving a subsequent auto promote request from the other node when in the auto promoted condition, a failure response is returned indicating that the other node is to cease providing host I/O access to the data storage object.
1. A method of providing a quorum service to a clustered pair of data storage nodes to protect against data corruption by failure of inter-node replication of write-type host I/O requests directed to a data storage object, comprising:
during an initial period of normal operation in which the data storage nodes each provide host I/O access to the data storage object and each replicate write-type requests to the other data storage node, registering each of the data storage nodes to the quorum service in response to respective registration requests therefrom; and
subsequently and based on both the data storage nodes being registered, (1) in response to receiving a first auto promote request from a first-requesting data storage node, returning a success response and entering an auto promoted condition, the success response indicating that the first-requesting data storage node is to continue providing the host I/O access to the data storage object without replicating the write-type host I/O requests to the other data storage node, and (2) in response to receiving a subsequent auto promote request from the other data storage node when in the auto promoted condition, returning a failure response indicating that the other data storage node is to cease providing host I/O access to the data storage object,
wherein the quorum service operates according to a state transition scheme including an Empty state in which no data storage nodes are registered, a 2-registered state in which both of the data storage nodes are registered, and an Auto Promoted state corresponding to the auto promoted condition, and wherein the Auto Promoted state is reached during normal operation from the 2-Registered state and not from the Empty state.
2. The method according to claim 1 , wherein the data storage object is one of a volume, volume group, file, file system, virtual volume, or container.
3. The method according to claim 1 , wherein the data storage nodes employ a time-to-live mechanism by which a preferred one of the data storage nodes regularly grants permission for ongoing operation to the other data storage node as a non-preferred node, and wherein the first auto promote request is sent by the first-requesting data storage node upon expiration of a time to live for the non-preferred node.
4. The method according to claim 3 , wherein the first-requesting data storage node is the preferred node, and the subsequent auto promote request is sent by the non-preferred node after a predetermined delay after expiration of the time to live.
5. The method according to claim 3 , wherein the first-requesting data storage node is the non-preferred node, and the first auto promote request is sent by the non-preferred node after a predetermined delay after expiration of the time to live.
6. The method according to claim 1 , wherein registering the data storage nodes includes receiving respective registration requests from the data storage nodes and registering each of the data storage nodes in response to its respective registration request.
7. The method according to claim 1 , wherein the quorum service is restored to the Auto Promoted state from the Empty state in response to receiving an auto promoted request from a data storage node that was auto promoted during normal operation preceding a loss of state at the quorum service.
8. The method according to claim 1 , wherein the state transition scheme further includes a Manual Promoted state reached in response to receiving a manual promote request from one of the data storage nodes based on action of an administrative user.
9. The method according to claim 1 , wherein the quorum service generates a quorum service identifier and provides it to the data storage nodes in response to identifier requests therefrom, the quorum service identifier being used by the data storage nodes to confirm mutual use of the same quorum service for protection of the data storage object.
10. The method according to claim 1 , wherein the state transition scheme further includes a 1-registered state in which only one of the data storage nodes is registered, and wherein the Auto Promoted state is reached during normal operation from the 2-Registered state and not from the 1-registered state.
11. A quorum service system comprising one or more computerized devices executing computer program instructions to cause the computerized devices to provide a quorum service to a clustered pair of data storage nodes to protect against data corruption by failure of inter-node replication of write-type host I/O requests directed to a data storage object, operation of the quorum service including:
During an initial period of normal operation in which the data storage nodes each provide host I/O access to the data storage object and each replicate write-type requests to the other data storage node, registering each of the data storage nodes to the quorum service in response to respective registration requests therefrom; and
subsequently and based on both the data storage nodes being registered, (1) in response to receiving a first auto promote request from a first-requesting data storage node, returning a success response and entering an auto promoted condition, the success response indicating that the first-requesting data storage node is to continue providing the host I/O access to the data storage object without replicating the write-type host I/O requests to the other data storage node, and (2) in response to receiving a subsequent auto promote request from the other data storage node when in the auto promoted condition, returning a failure response indicating that the other data storage node is to cease providing host I/O access to the data storage object,
wherein the quorum service operates according to a state transition scheme including an Empty state in which no data storage nodes are registered, a 2-registered state in which both of the data storage nodes are registered, and an Auto Promoted state corresponding to the auto promoted condition, and wherein the Auto Promoted state is reached during normal operation from the 2-Registered state and not from the Empty state.
12. The quorum service system according to claim 11 , wherein the data storage object is one of a volume, volume group, file, file system, virtual volume, or container.
13. The quorum service system according to claim 11 , wherein the data storage nodes employ a time-to-live mechanism by which a preferred one of the nodes regularly grants permission for ongoing operation to the other data storage node as a non-preferred node, and wherein the first auto promote request is sent by the first-requesting data storage node upon expiration of a time to live for the non-preferred node.
14. The quorum service system according to claim 13 , wherein the first-requesting data storage node is the preferred node, and the subsequent auto promote request is sent by the non-preferred node after a predetermined delay after expiration of the time to live.
15. The quorum service system according to claim 13 , wherein the first-requesting data storage node is the non-preferred node, and the first auto promote request is sent by the non-preferred node after a predetermined delay after expiration of the time to live.
16. The quorum service system according to claim 11 , wherein registering the data storage nodes includes receiving respective registration requests from the data storage nodes and registering each of the data storage nodes in response to its respective registration request.
17. The quorum service system according to claim 11 , wherein the quorum service is restored to the Auto Promoted state from the Empty state in response to receiving an auto promoted request from a data storage node that was auto promoted during normal operation preceding a loss of state at the quorum service.
18. The quorum service system according to claim 11 , wherein the state transition scheme further includes a Manual Promoted state reached in response to receiving a manual promote request from one of the data storage nodes based on action of an administrative user.
19. The quorum service system according to claim 11 , wherein the quorum service generates a quorum service identifier and provides it to the data storage nodes in response to identifier requests therefrom, the quorum service identifier being used by the data storage nodes to confirm mutual use of the same quorum service for protection of the data storage object.
20. The quorum service system according to claim 11 , wherein the state transition scheme further includes a 1-registered state in which only one of the data storage nodes is registered, and wherein the Auto Promoted state is reached during normal operation from the 2-Registered state and not from the 1-registered state.