Scalable storage system
Systems, methods, apparatuses, and software for data storage systems are provided herein. In one example, a data storage assembly is provided that includes a plurality of storage drives each comprising a PCIe host interface and solid state storage media. The data storage assembly includes a PCIe switch circuit coupled to the PCIe host interfaces of the storage drives and configured to receive storage operations issued by one or more host systems over a shared PCIe interface and transfer the storage operations for delivery to the storage drives over selected ones of the PCIe host interfaces. The data storage assembly includes a control processor configured to monitor usage statistics of the storage drives, and power control circuitry configured to selectively remove the power from ones of the storage drives based at least on the usage statistics of the storage drives.
1. A data storage system, comprising:
a plurality of storage modules each comprising one or more storage drives and a storage module Peripheral Component Interconnect Express (PCIe) interface;
a plurality of processing modules each comprising a processor and a processing module PCIe interface;
PCIe switch circuitry coupled to the storage module PCIe interfaces and the processing module PCIe interfaces and configured to establish logical isolation in the PCIe switch circuitry between one or more processors and one or more storage drives;
wherein individual processors among the plurality of processing modules instantiate selected storage drives among the plurality of storage modules for access to the selected storage drives over at least the logical isolation in the PCIe switch circuitry.
2. The data storage assembly of claim 1 , comprising:
a plurality of network modules each comprising a network interface and a network module PCIe interface coupled to the PCIe switch circuitry; and
the PCIe switch circuitry configured to establish second logical isolation in the PCIe switch circuitry between the individual processors and corresponding network modules.
3. The data storage assembly of claim 1 , comprising:
the plurality of processing modules each configured to:
receive storage operations into a corresponding processor, determine if the storage operations are directed to one or more storage drives managed by the corresponding processor, and when the storage operations are not directed to the one or more storage drives managed by the corresponding processor, then the corresponding processor responsively transfers associated storage operations over the PCIe switch circuitry to another processor that manages one or more storage drives indicated by the associated storage operations.
4. The data storage assembly of claim 1 , comprising:
the plurality of processing modules each configured to:
receive storage operations into a corresponding processor, determine if the storage operations are directed to one or more storage drives managed by the corresponding processor, and when the storage operations are directed to the one or more storage drives managed by the corresponding processor, then the corresponding processor responsively transfers associated storage operations over the PCIe switch circuitry to the one or more storage drives.
5. The data storage assembly of claim 4 , wherein the corresponding processor responsively transfers the associated storage operations over the PCIe switch circuitry to stripe corresponding data over the one or more storage drives.
6. The data storage assembly of claim 1 , comprising:
a plurality of network modules each comprising a network interface and a network module PCIe interface;
the PCIe switch circuitry further coupled to the network module PCIe interfaces; and
individual network interfaces configured to communicate over the PCIe switch circuitry only with assigned processors.
7. The data storage assembly of claim 6 , comprising:
wherein any of the network interfaces are configured to receive storage operations directed to any of the processors of the data storage system.
8. The data storage assembly of claim 7 , comprising:
the plurality of processing modules each configured to:
receive storage operations into a corresponding processor from a corresponding network module coupled by the PCIe switch circuitry, determine if the storage operations are directed to one or more storage drives managed by the corresponding processor, and when the storage operations are not directed to the one or more storage drives managed by the corresponding processor, then the corresponding processor responsively transfers associated storage operations over the PCIe switch circuitry to another processor that manages one or more storage drives indicated by the associated storage operations.
9. The data storage assembly of claim 7 , wherein the network interfaces receive the storage operations in an iSCSI protocol from external devices coupled over associated network links.
10. A method of operating a data storage system, comprising:
in Peripheral Component Interconnect Express (PCIe) switch circuitry, communicatively coupling to storage module PCIe interfaces for a plurality of storage modules, wherein each of the plurality of storage modules comprise one or more storage drives and a storage module PCIe interface;
in the PCIe switch circuitry, communicatively coupling to processing module PCIe interfaces for a plurality of processing modules, wherein each of the plurality of processing modules comprise a processor and a processing module PCIe interface;
in the PCIe switch circuitry, establishing logical isolation in the PCIe switch circuitry between one or more processors and one or more storage drives;
wherein individual processors among the plurality of processing modules instantiate selected storage drives among the plurality of storage modules for access to the selected storage drives over at least the logical isolation in the PCIe switch circuitry.
11. The method of claim 10 , comprising:
in the PCIe switch circuitry, communicatively coupling to network module PCIe interfaces for a plurality of network modules, wherein each of the plurality of network modules comprise a network interface; and
in the PCIe switch circuitry, establishing second logical isolation in the PCIe switch circuitry between the individual processors and corresponding network modules.
12. The method of claim 10 , comprising:
in each of the plurality of processing modules, receiving storage operations into a corresponding processor, determining if the storage operations are directed to one or more storage drives managed by the corresponding processor, when the storage operations are not directed to the one or more storage drives managed by the corresponding processor, then in the corresponding processor, responsively transferring associated storage operations over the PCIe switch circuitry to another processor that manages one or more storage drives indicated by the associated storage operations.
13. The method of claim 10 , comprising:
in each of the plurality of processing modules receiving storage operations into a corresponding processor, determining if the storage operations are directed to one or more storage drives managed by the corresponding processor, and when the storage operations are directed to the one or more storage drives managed by the corresponding processor, then in the corresponding processor, responsively transferring associated storage operations over the PCIe switch circuitry to the one or more storage drives.
14. The method of claim 13 , wherein the corresponding processor responsively transfers the associated storage operations over the PCIe switch circuitry to stripe corresponding data over the one or more storage drives.
15. The method of claim 10 , comprising:
in the PCIe switch circuitry, further communicatively coupling to network module PCIe interfaces for a plurality of network modules, wherein each of the plurality of network modules comprise a network interface and a network module PCIe interface; and
in the PCIe switch circuitry, configuring individual network interfaces to communicate over the PCIe switch circuitry only with assigned processors.
16. The method of claim 15 , comprising:
wherein any of the network interfaces are configured to receive storage operations directed to any of the processors of the data storage system.
17. The method of claim 16 , comprising:
in each of the plurality of processing modules, receiving storage operations into a corresponding processor from a corresponding network module coupled by the PCIe switch circuitry, determining if the storage operations are directed to one or more storage drives managed by the corresponding processor, when the storage operations are not directed to the one or more storage drives managed by the corresponding processor, then in the corresponding processor, responsively transferring associated storage operations over the PCIe switch circuitry to another processor that manages one or more storage drives indicated by the associated storage operations.
18. The method of claim 17 , wherein the network interfaces receive the storage operations in an iSCSI protocol from external devices coupled over associated network links.
19. A data storage system, comprising:
a plurality of storage modules each comprising one or more storage drives and a storage module Peripheral Component Interconnect Express (PCIe) interface;
a plurality of processing modules each comprising a processor and a processing module PCIe interface;
a plurality of network modules each comprising a network interface and a network module PCIe interface;
PCIe switch circuitry coupled to the storage module PCIe interfaces, the processing module PCIe interfaces, and the network module PCIe interfaces, and configured to:
establish first logical isolation in the PCIe switch circuitry between the individual processors and corresponding instantiated storage drives; and
establish second logical isolation in the PCIe switch circuitry between the individual processors and corresponding network module PCIe interfaces;
wherein individual processors among the plurality of processing modules instantiate selected storage drives among the plurality of storage modules for access to the selected storage drives over at least the first logical isolation in the PCIe switch circuitry.
20. The data storage system of claim 19 , wherein the plurality of processing modules are each configured to:
receive storage operations into a corresponding processor from a corresponding network module using at least the second logical isolation in the PCIe switch circuitry, and determine if the storage operations are directed to one or more storage drives managed by the corresponding processor;
when the storage operations are not directed to the one or more storage drives managed by the corresponding processor, then the corresponding processor responsively transfers associated storage operations to another processor that manages one or more storage drives indicated by the associated storage operations using at least the first logical isolation in the PCIe switch circuitry; and
when the storage operations are directed to the one or more storage drives managed by the corresponding processor, then the corresponding processor responsively transfers associated storage operations to the one or more storage drives using at least the first logical isolation in the PCIe switch circuitry.