Failover handling in modular switched fabric for data storage systems
Data storage systems are provided that include network interfaces each configured to receive storage operations issued by one or more host systems for handling by the data storage system, and transfer the storage operations over a Peripheral Component Interconnect Express (PCIe) fabric. The data storage systems also include processors configured to receive the storage operations over the PCIe fabric, process properties of the storage operations against allocation information of a shared PCIe address space to determine target processors to handle the storage operations, and selectively transfer ones of the storage operations to the target processors for handling of the storage operations with associated storage drives coupled over the PCIe fabric. The data storage systems also identifie failed ones of the processors and responsively initiate backup ones of the processors to handle portions of the shared PCIe address space handled by the failed ones of the processors.
1. A data storage system, comprising:
a plurality of network interfaces each configured to receive storage operations issued by one or more host systems for handling by the data storage system, and transfer the storage operations over a Peripheral Component Interconnect Express (PCIe) fabric;
a plurality of processors configured to receive the storage operations over the PCIe fabric, process properties of the storage operations against allocation information of a shared PCIe address space to determine target processors to handle the storage operations, and selectively transfer ones of the storage operations to the target processors for handling of the storage operations with associated storage drives coupled over the PCIe fabric, wherein the shared PCIe address space is shared among at least the plurality of processors on the PCIe fabric; and
a monitoring system configured to identify failed ones of the processors and responsively initiate backup ones of the processors to handle portions of the shared PCIe address space handled by the failed ones of the processors.
2. The data storage system of claim 1 , comprising:
the plurality of processors configured to maintain failover tables comprising indications of the allocation information of the shared PCIe address space among the plurality of processors.
3. The data storage system of claim 1 , comprising:
the plurality of processors comprising the monitoring system, and configured to periodically poll one or more monitored processors to identify failed ones of the processors.
4. The data storage system of claim 1 , comprising:
a control system configured to subdivide the shared PCIe address space to allocate a plurality of storage drives among individual processors, wherein each subdivision of the shared PCIe address space corresponds to an associated subset of the plurality of storage drives.
5. The data storage system of claim 1 , comprising:
a control system configured to assign each of a plurality of storage drives an associated unique identifier and report the associated unique identifiers to the processors for handling of the storage operations over the PCIe fabric.
6. The data storage system of claim 1 , comprising:
a control system configured to initialize a plurality of storage drives for use on the PCIe fabric by the plurality of processors;
the control system configured to assign each of the plurality of storage drives an associated unique identifier using at least a sideband communication link distinct from the PCIe fabric, the sideband communication link communicatively coupling the control system and the plurality of storage drives; and
the control system configured to report the associated unique identifiers to the plurality of processors.
7. The data storage system of claim 1 , comprising:
responsive to a failed processor, a monitoring processor configured to inherit network connection states from the failed processor and accept ones of the storage operations directed to a network interface initially assigned to the failed processor.
8. The data storage system of claim 1 , comprising:
responsive to a failed processor, a monitoring processor configured to initiate a rediscovery process over the PCIe fabric with one or more storage drives formerly managed by the failed processor to transfer management to the monitoring processor.
9. The data storage system of claim 1 , comprising:
responsive to detecting the failed ones of the processors, the monitoring system configured to restart the failed ones of the processors.
10. A method of operating a data storage system, the method comprising:
in a plurality of network interfaces, receiving storage operations issued by one or more host systems for handling by the data storage system, and transferring the storage operations over a Peripheral Component Interconnect Express (PCIe) fabric;
in a plurality of processors, receiving the storage operations over the PCIe fabric, processing properties of the storage operations against allocation information of a shared PCIe address space to determine target processors to handle the storage operations, and selectively transferring ones of the storage operations to the target processors for handling of the storage operations with associated storage drives coupled over the PCIe fabric, wherein the shared PCIe address space is shared among at least the plurality of processors on the PCIe fabric; and
in a monitoring system, identifying failed ones of the processors and responsively initiating backup ones of the processors to handle portions of the shared PCIe address space handled by the failed ones of the processors.
11. The method of claim 10 , further comprising:
in the plurality of processors, maintaining failover tables comprising indications of the allocation information of the shared PCIe address space among the plurality of processors.
12. The method of claim 10 , further comprising:
the plurality of processors, periodically polling one or more monitored processors to identify failed ones of the processors.
13. The method of claim 10 , further comprising:
in a control system, subdividing the shared PCIe address space to allocate a plurality of storage drives among individual processors, wherein each subdivision of the shared PCIe address space corresponds to an associated subset of the plurality of storage drives.
14. The method of claim 10 , further comprising:
in a control system, assigning each of a plurality of storage drives an associated unique identifier, and reporting the associated unique identifiers to the processors for handling of the storage operations over the PCIe fabric.
15. The method of claim 10 , further comprising:
in a control system, initializing a plurality of storage drives for use on the PCIe fabric by the processors, and assigning each of the plurality of storage drives an associated unique identifier using at least a sideband communication link distinct from the PCIe fabric, the sideband communication link communicatively coupling the control system and the plurality of storage drives; and
in the control system, reporting the associated unique identifiers to the plurality of processors.
16. The method of claim 10 , further comprising:
responsive to a failed processor, in a monitoring processor, inheriting network connection states from the failed processor and accepting ones of the storage operations directed to a network interface initially assigned to the failed processor.
17. The method of claim 10 , further comprising:
responsive to a failed processor, in a monitoring processor, initiating a rediscovery process over the PCIe fabric with one or more storage drives formerly managed by the failed processor to transfer management to the monitoring processor.
18. The method of claim 10 , further comprising:
responsive to detecting the failed ones of the processors, restarting the failed ones of the processors.
19. A modular data storage system, comprising:
a plurality of network modules each configured to receive storage operations issued by one or more host systems for handling by the data storage system, and transfer the storage operations over a Peripheral Component Interconnect Express (PCIe) fabric;
a plurality of processing modules each comprising one or more processors configured to receive the storage operations over the PCIe fabric, process properties of the storage operations against allocation information of a shared PCIe address space to determine target processors to handle the storage operations, and selectively transfer ones of the storage operations to the target processors for handling of the storage operations with associated storage drives coupled over the PCIe fabric, wherein the shared PCIe address space is shared among at least the plurality of processors on the PCIe fabric;
a plurality of storage modules each comprising a plurality of storage drives communicatively coupled to the PCIe fabric, and
monitoring ones of the processors configured to identify failed ones of the processors and responsively initiate backup ones of the processors to handle portions of the shared PCIe address space formerly handled by the failed ones of the processors.
20. The modular data storage system of claim 19 , comprising:
responsive to a failed processor, a monitoring processor configured to:
inherit network connection states from the failed processor;
accept ones of the storage operations directed to a network interface initially assigned to the failed processor; and
initiate a rediscovery process over the PCIe fabric with one or more storage drives formerly managed by the failed processor to transfer management to the monitoring processor.