Failover handling in modular switched fabric for data storage systems
Systems, methods, apparatuses, and software for data storage systems are provided herein. In one example, a data storage system is provided that includes storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured to store data and retrieve the data stored on associated storage media responsive to data transactions. The data storage system also includes processing modules each comprising one or more processors, where a first processor is configured to receive data transactions and transfer the data transactions for delivery to associated ones of the storage drives. A monitoring system is configured to identify when the first processor fails, and responsively instruct a second processor to handle further data transactions directed to storage drives associated with the first processor.
1. A data storage system, comprising:
a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured to store data and retrieve the data stored on associated storage media responsive to storage operations;
a plurality of processing modules each comprising one or more processors, where a first processor is configured to receive at least a portion of the storage operations and transfer the portion of the storage operations for delivery to associated ones of the plurality of storage drives over a PCIe interface;
a control processor configured to initialize the plurality of storage drives for use on the PCIe interfaces by the processors on the processing modules, assign each of the plurality of storage drives an associated unique identifier using at least a sideband communication link distinct from the PCIe interfaces, wherein the sideband communication link communicatively couples the control processor and the ones of the plurality of storage drives, and report the associated unique identifiers to the processing modules over at least an associated interface; and
a monitoring system configured to identify when the first processor fails, and responsively instruct a second processor to handle further storage operations directed to storage drives associated with the first processor.
2. The data storage system of claim 1 , wherein the first processor and the second processor are both included in a first processing module.
3. The data storage system of claim 1 , wherein the first processor is included in a first processing module and the second processor is included in a second processing module.
4. The data storage system of claim 1 , comprising:
the control processor further configured to subdivide the plurality of storage drives among individual processors of the processing modules so that each of the individual processors manages ones of the storage operations directed to an associated subset of the plurality of storage drives.
5. The data storage system of claim 1 , comprising:
the control processor further configured to subdivide a shared PCIe address space to subdivide the plurality of storage drives among individual processors of the processing modules, wherein each subdivision of the shared PCIe address space corresponds to an associated subset of the plurality of storage drives.
6. The data storage system of claim 5 , wherein the first processor is configured to handle the portion of the storage operations for a first subset of the plurality of storage drives that corresponds to a first subdivision of the shared PCIe address space; and comprising:
the monitoring system configured to instruct the second processor to handle the further storage operations directed to the storage drives associated with the first processor by at least indicating to the second processor that the second processor is to handle the further storage operations for the first subset of the plurality of storage drives that corresponds to the first subdivision of the shared PCIe address space.
7. The data storage system of claim 1 , comprising:
the second processor configured to maintain a failover table comprising indications of portions of a shared PCIe address space among the processing modules, wherein a first portion corresponds to a subset of the plurality of storage drives managed by the first processor and a second portion corresponds to a subset of the plurality of storage drives managed by the second processor;
responsive to identifying when the first processor fails, the second processor configured to manage the first portion the shared PCIe address space instead of the first processor.
8. The data storage system of claim 1 , wherein the second processor comprises the monitoring system, and comprising:
the second processor configured to periodically poll the first processor to identify when the first processor fails.
9. A method of operating a data storage system, the method comprising:
in a plurality of storage drives, storing data and retrieving the data stored on associated storage media responsive to storage operations received over a Peripheral Component Interconnect Express (PCIe) interface;
in a first processor among a plurality of processing modules each comprising one or more processors, receiving at least a portion of the storage operations and transferring the portion of the storage operations for delivery to associated ones of the plurality of storage drives over a PCIe interface;
initializing the plurality of storage drives for use on the PCIe interfaces by the processors on the processing modules;
assigning each of the plurality of storage drives an associated unique identifier using at least a sideband communication link distinct from the PCIe interfaces, the sideband communication link communicatively coupling a control processor and the ones of the plurality of storage drives;
reporting the associated unique identifiers to the processing modules over at least an Ethernet interface; and
responsive to identifying when the first processor fails instructing a second processor to handle further storage operations directed to storage drives associated with the first processor.
10. The method of claim 9 , wherein the first processor and the second processor are both included in a first processing module.
11. The method of claim 9 , wherein the first processor is included in a first processing module and the second processor is included in a second processing module.
12. The method of claim 9 , further comprising:
subdividing the plurality of storage drives among individual processors of the processing modules so that each of the individual processors manages ones of the storage operations directed to an associated subset of the plurality of storage drives.
13. The method of claim 9 , further comprising:
establishing a shared PCIe address space to subdivide the plurality of storage drives among individual processors of the processing modules, wherein each subdivision of the shared PCIe address space corresponds to an associated subset of the plurality of storage drives.
14. The method of claim 13 , wherein the first processor is configured to handle the portion of the storage operations for a first subset of the plurality of storage drives that corresponds to a first subdivision of the shared PCIe address space; and further comprising:
instructing the second processor to handle the further storage operations directed to the storage drives associated with the first processor by at least indicating to the second processor that the second processor is to handle the further storage operations for the first subset of the plurality of storage drives that corresponds to the first subdivision of the shared PCIe address space.
15. The method of claim 9 , further comprising:
in the second processor, maintaining a failover table comprising indications of portions of a shared PCIe address space among the processing modules, wherein a first portion corresponds to a subset of the plurality of storage drives managed by the first processor and a second portion corresponds to a subset of the plurality of storage drives managed by the second processor;
in the second processor, responsive to identifying when the first processor fails, managing the first portion the shared PCIe address space instead of the first processor.
16. The method of claim 9 , wherein the second processor comprises the monitoring system, and further comprising:
in the second processor periodically polling the first processor to identify when the first processor fails.
17. A modular data storage system, comprising:
a plurality of storage modules each comprising a plurality of solid state drives and configured to communicate over an associated Peripheral Component Interconnect Express (PCIe) interface;
a plurality of processing modules each comprising one or more processors that are configured to manage storage operations for a designated subset of the plurality of storage modules over associated PCIe interfaces;
a control processor configured to initialize the plurality of solid state drives for use on the associated PCIe interfaces by one or more of the processors on the processing modules, assign each of the plurality of solid state drives an associated unique identifier using at least a sideband communication link distinct from the associated PCIe interfaces, wherein the sideband communication link communicatively couples the control processor and ones of the plurality of solid state drives, and report the associated unique identifiers to the processing modules over at least an associated interface;
each of the processors configured to monitor one or more of the processors to detect failures among the one or more of the processors; and
responsive to detecting a failed processor, a monitoring one of the processors configured to take over storage operations for the failed processor by at least managing storage operations for a designated subset of the plurality of storage modules associated with the failed processor.
18. The modular data storage system of claim 17 , comprising:
the monitoring one of the processors configured to maintain a failover table indicating at least the designated subset of the plurality of storage modules associated with the failed processor.