Method and apparatus for node-pair enclosure replacement

1 . A method comprising:

providing a first enclosure having a first storage node coupled to a second storage node by a first node-to-node interconnect externally of the first enclosure, wherein the first storage node and the second storage node communicate independently of the first enclosure;

providing a second enclosure having a third storage node coupled to a fourth storage node by a second node-to-node interconnect externally of the second enclosure, wherein the third storage node and the fourth storage node communicate independently of the second enclosure;

providing a network coupling each of the first storage node and the second storage node of the first enclosure to the third storage node and the fourth storage node of the second enclosure;

monitoring, by a storage array system, the first enclosure for a fault condition;

detecting, based on the monitoring, a node enclosure fault of the first enclosure; and

responsive to detecting the node enclosure fault,

replacing the first enclosure without interrupting communication between the first storage node and the second storage node.

2 . The method of claim 1 wherein replacing the first enclosure comprises:

disabling the first storage node and disconnecting the first node-to-node interconnect to the second storage node;

moving the first storage node to a third enclosure;

coupling the first storage node in the third enclosure to the second storage node in the first enclosure with the first node-to-node interconnect and enabling the first storage node; and

performing a first sync of the first storage node and the second storage node.

3 . The method of claim 2 further comprising:

disabling the second storage node in the first enclosure and disconnecting the first node-to-node interconnect to the first storage node in the second enclosure;

moving the second storage node to a fourth enclosure;

coupling the second storage node in the fourth enclosure to the first storage node in the third enclosure with the first mode-to-mode interconnect and enabling the second storage node;

performing a second sync of the second storage node in the fourth enclosure with the first storage node in the third enclosure;

disabling the first storage node in the third enclosure and disconnecting the first node-to-node interconnect to the second storage node;

moving the first storage node to the fourth enclosure;

coupling the first storage node in the fourth enclosure to the second storage node with the first node-to-node interconnect and enabling the first storage node; and

performing a third sync of the first storage node and the second storage node.

4 . The method of claim 3 wherein performing the first sync, the second sync and the third sync include mirroring the first storage node and the second storage node.

5 . A system comprising:

a first node enclosure housing a first storage node and a second storage node;

a second node enclosure coupled to the first node enclosure by a network;

a first node-to-node interconnect coupling the first storage node to the second storage node externally of the first node enclosure;

a memory; and

at least one processor that is operatively coupled to the memory, the at least one processor being configured to perform the operations of:

monitoring the first enclosure for a fault condition;

detecting, based on the monitoring, a node enclosure fault of the first enclosure,

wherein the first storage node and second storage node are configured to communicate independently of the first node enclosure and the first node enclosure, upon detection of the node enclosure fault, is replaceable without disruption of communication over the network.

6 . The system of claim 5 wherein the second node enclosure houses a third storage node and a fourth storage node coupled by a second node-to-node interconnect externally of the second node enclosure, and wherein the third storage node and fourth node are configured to communicate independent of the second node enclosure.

7 . The system of claim 5 wherein the network is an InfiniBand network.

8 . The system of claim 5 wherein the network is an unswitched network.

9 . The system of claim 5 wherein the first node-to-node interconnect comprises a peripheral component interconnect express (PCIe) interface.

10 . The system of claim 5 wherein the first storage node comprises a first host channel adapter coupled to the network and the second storage node comprises a second host channel adapter coupled to the network.

11 . The system of claim 5 wherein the first storage node comprises a first peripheral interconnect card configured to receive the first node-to-node interconnect and the second storage node comprises a second peripheral interconnect card configured to receive the first node-to-node interconnect.

12 . The system of claim 5 wherein the first storage node and the second storage node are mirrored storage devices.

13 . The system of claim 5 wherein the first node-to-node interconnect comprises redundant connections to the first storage node and the second storage node.

14 . A storage system comprising:

a first node enclosure housing a first storage node and a second storage node;

a first node-to-node interconnect coupling the first storage node to the second storage node externally of the first node enclosure;

a second node enclosure housing a third storage node and a fourth storage node;

a second node-to-node interconnect coupling the third storage node to the fourth storage node externally of the second node enclosure;

an unswitched network coupling the first storage node to the third storage node and the fourth storage node and coupling the second storage node to the third storage node and fourth storage node;

a memory; and

at least one processor that is operatively coupled to the memory, the at least one processor being configured to perform the operations of:

monitoring the first enclosure and the second enclosure for a fault condition;

detecting, based on the monitoring, a node enclosure fault of one of the first enclosure and the second enclosure,

wherein the first storage node is configured to communicate with the second storage node independently of the first node enclosure, the third storage node is configured to communicate with the fourth storage node independently of the second node enclosure, and one of the first node enclosure or the second node enclosure, upon detection of the node enclosure fault, is replaceable without disruption of communication over the unswitched network.

15 . The storage system of claim 14 wherein the unswitched network is an InfiniBand network.

16 . The storage system of claim 14 wherein the first node-to-node interconnect and the second interconnect comprise peripheral component interconnect express (PCIe) interfaces.

17 . The storage system of claim 14 wherein each of the first storage node, the second storage node, the third storage node and the fourth storage node comprises a host channel adapter coupled to the unswitched network.

18 . The storage system of claim 14 wherein each of the first storage node and the second storage node, comprises peripheral interconnect cards configured to receive the first node-to-node interconnect and each of the third storage node and the fourth storage node comprises peripheral interconnect cards configured to receive the second node-to-node interconnect.

19 . The storage system of claim 14 wherein the first storage node is configured to mirror the second storage node and the third storage node is configured to mirror the fourth storage node.

20 . The storage system of claim 14 wherein the first node-to-node interconnect includes redundant connections to the first storage node and the second storage node, and the second node-to-node interconnect includes redundant connections to the third storage node and the fourth storage node.