Distributed State Machine for High Availability of Non-Volatile Memory in Cluster Based Computing Systems
Highly available storage clusters, including NVMeoF storage clusters are disclosed. Such high availability storage cluster may have increased sized, higher availability and decreased individual hardware requirements
1 . A high availability (HA) storage cluster, comprising
a switch;
a set of storage resources coupled to a set of hosts and coupled to the switch via a first network; and
a set of nodes coupled via a second network to the switch, each node including an HA module thereon, the HA modules at each of the set of nodes cooperating to maintain a state for the HA storage cluster,
each HA module at each node maintaining the state of the HA cluster by communicating with each of the other HA modules to synchronize the state maintained at each of the other HA modules with the state maintained at the HA module at that node and a state machine for the HA cluster, wherein the state includes an association between each of the resources and a corresponding one of the set of nodes wherein an HA module at a first node is adapted for:
monitoring resources of the HA storage cluster;
detecting a failure in a resource of a storage cluster;
accessing the state maintained at the first node to determine that the switch is programmed such that the failed resource is assigned to a partition of the switch associated with the failed resource;
determining a second node to which the failed resource should be assigned based on the state machine of the HA module at the first node and the state maintained at the first node; and
reprogramming, by the HA module at the first node, the switch such that the switch is reconfigured to assign the storage resource and the second node to a same virtual switch partition
2 . The system of claim 1 , wherein the resources include the switch, the set of nodes, the set of storage resources, a volume, a Logical Unit Number (LUN) or a network path.
3 . The system of claim 1 , wherein monitoring resources includes attempting to access the resources at a time interval.
4 . The system of claim 1 , wherein monitoring resources includes communicating a heartbeat message between each HA module.
5 . The system of claim 1 , wherein the failure is a bandwidth or data rate associated with the resource falling below a certain threshold level.
6 . The system of claim 1 , wherein the first network and the second network are the same network.
7 . The system of claim 1 , wherein the switch is a PCI Express switch.
8 . A method for operating a high availability (HA) storage cluster, comprising
providing a set of HA modules at a set of nodes of an HA storage cluster, the HA storage cluster including a set of storage resources coupled to a set of hosts and coupled to a switch via a first network, wherein the set of nodes are coupled via a second network to the switch;
cooperating between the HA module at the set of nodes to maintain a state for the HA storage cluster, each HA module at each node maintaining the state of the HA cluster by communicating with each of the other HA modules to synchronize the state maintained at each of the other HA modules with the state maintained at the HA module at that node and a state machine for the HA cluster, wherein the state includes an association between each of the resources and a corresponding one of the set of nodes, and at an HA module of a first node:
monitoring resources of the HA storage cluster;
detecting a failure in a resource of a storage cluster;
accessing the state maintained at the first node to determine that the switch is programmed such that the failed resource is assigned to a partition of the switch associated with the failed resource;
determining a second node to which the failed resource should be assigned based on the state machine of the HA module at the first node and the state maintained at the first node; and
reprogramming, by the HA module at the first node, the switch such that the switch is reconfigured to assign the storage resource and the second node to a same virtual switch partition
9 . The method of claim 8 , wherein the resources include the switch, the set of nodes, the set of storage resources, a volume, a Logical Unit Number (LUN) or a network path.
10 . The method of claim 8 , wherein monitoring resources includes attempting to access the resources at a time interval.
11 . The method of claim 8 , wherein monitoring resources includes communicating a heartbeat message between each HA module.
12 . The method of claim 8 , wherein the failure is a bandwidth or data rate associated with the resource falling below a certain threshold level.
13 . The method of claim 8 , wherein the first network and the second network are the same network.
14 . The method of claim 8 , wherein the switch is a PCI Express switch.
15 . A non-transitory computer readable medium comprising instruction for:
providing a set of HA modules at a set of nodes of an HA storage cluster, the HA storage cluster including a set of storage resources coupled to a set of hosts and coupled to a switch via a first network, wherein the set of nodes are coupled via a second network to the switch;
cooperating between the HA module at the set of nodes to maintain a state for the HA storage cluster, each HA module at each node maintaining the state of the HA cluster by communicating with each of the other HA modules to synchronize the state maintained at each of the other HA modules with the state maintained at the HA module at that node and a state machine for the HA cluster, wherein the state includes an association between each of the resources and a corresponding one of the set of nodes, and at an HA module of a first node:
monitoring resources of the HA storage cluster;
detecting a failure in a resource of a storage cluster;
accessing the state maintained at the first node to determine that the switch is programmed such that the failed resource is assigned to a partition of the switch associated with the failed resource;
determining a second node to which the failed resource should be assigned based on the state machine of the HA module at the first node and the state maintained at the first node; and
reprogramming, by the HA module at the first node, the switch such that the switch is reconfigured to assign the storage resource and the second node to a same virtual switch partition
16 . The non-transitory computer readable medium of claim 15 , wherein the resources include the switch, the set of nodes, the set of storage resources, a volume, a Logical Unit Number (LUN) or a network path.
17 . The non-transitory computer readable medium of claim 15 , wherein monitoring resources includes attempting to access the resources at a time interval.
18 . The non-transitory computer readable medium of claim 15 , wherein monitoring resources includes communicating a heartbeat message between each HA module.
19 . The non-transitory computer readable medium of claim 15 , wherein the failure is a bandwidth or data rate associated with the resource falling below a certain threshold level.
20 . The non-transitory computer readable medium of claim 15 , wherein the first network and the second network are the same network.
21 . The non-transitory computer readable medium of claim 15 , wherein the switch is a PCI Express switch.