Storage engine having compute nodes with redundant fabric access
A storage system includes a storage engine having a first compute node, a second compute node, a first fabric adapter, and a second fabric adapter, the first compute node having a first memory and the second compute node having a second memory. The first compute node is connected to both the first and second fabric adapters, and the second compute node is connected to both the second and first fabric adapters. Both fabric adapters are configured to perform atomic operations on a memory of its respective compute node, and each fabric adapter contains a multi-initiating module configured to enable both the first compute node and the second compute node to initiate memory access operations on its respective memory.
1. A storage system, comprising:
a first storage engine having a first compute node, a second compute node, a first fabric access module, and a second fabric access module, the first compute node having a first CPU and associated first memory and the second compute node having a second CPU and associated second memory;
a first connection within the first storage engine between the first compute node and the first fabric access module;
a second connection within the first storage engine between the first compute node and the second fabric access module;
a third connection within the first storage engine between the second compute node and the first fabric access module; and
a fourth connection within the first storage engine between the second compute node and the second fabric access module;
wherein the first fabric access module contains a first multi-initiator module configured to enable both the first compute node and the second compute node to directly initiate atomic memory access operations on the first memory through the first fabric access module; and
wherein the second host adapter contains a second multi-initiator module configured to enable both the second compute node and the first compute node to directly initiate atomic memory access operations on the second memory through the second fabric access module.
2. The storage system of claim 1 , wherein the first fabric access module contains a first host adapter configured to perform atomic operations on the first memory and the second memory; and
wherein the second fabric access module contains a second host adapter configured to perform atomic operations on the second memory and the first memory.
3. The storage system of claim 1 , wherein the first fabric access module has a first upstream PCIe port connected to a first PCIe root complex associated with the first CPU and first memory of the first compute node;
wherein the first fabric access module has a second upstream PCIe port connected to a second PCIe root complex associated with the second CPU and second memory of the second compute node;
wherein the second fabric access module has a third upstream PCIe port connected to the second PCIe root complex associated with the second CPU and second memory of the second compute node; and
wherein the second fabric access module has a fourth upstream PCIe port connected to the first PCIe root complex associated with the first CPU and first memory of the first compute node.
4. The storage system of claim 1 , wherein the first fabric access module includes a first PCIe interface connected to the first connection, a third PCIe interface connected to the third connection, and a first host adapter configured to enable the fabric access module to interconnect with a second storage engine; and
wherein the second fabric access module includes a second PCIe interface connected to the second connection, a fourth PCIe interface connected to the fourth connection, and a second host adapter configured to enable the fabric access module to interconnect with a second storage engine.
5. The storage system of claim 1 , further comprising:
a second storage engine having a third compute node, a fourth compute node, a third fabric access module, and a fourth fabric access module, the third compute node having a third memory and the fourth compute node having a fourth memory;
a fifth connection between the third compute node and the third fabric access module;
a sixth connection between the third compute node and the fourth fabric access module;
a seventh connection between the fourth compute node and the fourth fabric access module; and
an eighth connection between the fourth compute node and the third fabric access module.
6. The storage system of claim 5 , wherein the first fabric access module contains a first multi-initiator module configured to enable both the first compute node and the second compute node to initiate memory access operations on the third memory of the third compute node and the fourth memory of the fourth compute node through the first fabric access module; and
wherein the second host adapter contains a second multi-initiator module configured to enable both the second compute node and the first compute node to initiate memory access operations on the third memory of the third compute node and the fourth memory of the fourth compute node through the second fabric access module.
7. The storage system of claim 6 , wherein the first multi-initiator module is further configured to enable both the third compute node and the fourth compute node to initiate memory access operations on the first memory of the first compute node and the second memory of the second compute node through the first fabric access module; and
wherein the second multi-initiator module is further configured to enable both the third compute node and the fourth compute node to initiate memory access operations on the second memory of the second compute node and the first memory of the first compute node through the second fabric access module.
8. The storage system of claim 6 , wherein the first multi-initiator module is further configured to enable both the third compute node and the fourth compute node to initiate atomic memory operations on the first memory of the first compute node and the second memory of the second compute node through the first fabric access module; and
wherein the second multi-initiator module is further configured to enable both the third compute node and the fourth compute node to initiate atomic memory operations on the second memory of the second compute node and the first memory of the first compute node through the second fabric access module.
9. The storage system of claim 5 , wherein the first fabric access module contains a first multi-initiator module configured to enable both the first compute node and the second compute node to initiate atomic memory operations on the third memory of the third compute node and the fourth memory of the fourth compute node through the first fabric access module; and
wherein the second host adapter contains a second multi-initiator module configured to enable both the second compute node and the first compute node to initiate atomic memory operations on the third memory of the third compute node and the fourth memory of the fourth compute node through the second fabric access module.