Methods and structure for improved I/O shipping in a clustered storage system
Methods and structure for improved shipping of I/O requests among multiple storage controllers of a clustered storage system. Minimal processing of a received I/O request is performed in a first controller to determine whether the I/O request is directed to a logical volume that is owned by the first controller or to a logical volume owned by another controller. For requests to logical volumes owned by another controller, the original I/O request is modified to indicate the target device address of the other controller. The first controller then ships the request to the other controller and configures DMA capabilities of the first controller to exchange data associated with the shipped request between the other controller and memory of the host system.
1. A clustered storage system comprising:
a first storage controller adapted to receive an I/O request from an attached host system and adapted to couple with a plurality of physical storage devices on which a logical volume is configured; and
a second storage controller coupled with the first storage controller and coupled with the plurality of storage devices, wherein the second storage controller owns the logical volume,
wherein the first storage controller is adapted to detect that the received I/O request is directed to the logical volume and is further adapted to transmit the received I/O request as a modified I/O request to the second storage controller responsive to the detection, wherein the modified I/O request comprises a modified destination address corresponding to the second storage controller,
wherein the second storage controller is adapted to process the modified I/O request by accessing affected storage devices of the plurality of physical storage devices based on a configuration of the logical volume, and
wherein the second storage controller exchanges data associated with the modified I/O request with the host system through the first storage controller.
2. The system of claim 1
wherein the second storage controller is adapted to return a completion status to the first storage controller indicating a status of completion of processing of the modified I/O request, and
wherein the first storage controller is adapted to return the received completion status to the attached host system.
3. The system of claim 1
wherein the first storage controller comprises a first inter-controller interface circuit for coupling with the second storage controller, the first inter-controller interface circuit having direct memory access (DMA) capability for access to memory of the host system,
wherein the first storage controller is adapted to generate the modified I/O request by copying the received I/O request and modifying a destination address in the modified I/O request to a destination address of the second storage controller, wherein the received I/O request comprises DMA scatter-gather list indicating addresses in a memory of the host system for data associated with the received I/O request, and
wherein the first storage controller is adapted to utilize the DMA scatter-gather list with the DMA capability of the first inter-controller interface circuit to transfer data relating to the modified I/O request between the memory of the host system and the second storage controller.
4. The system of claim 3
wherein the first storage controller comprises a control circuit for managing operation of the first storage controller, the control circuit coupled with the first inter-controller interface circuit,
wherein the control circuit is adapted to configure the DMA capability of the first inter-controller interface circuit to transfer data relating to the modified I/O request between the memory of the host system and the second storage controller, wherein the data is transferred in accordance with the scatter-gather list.
5. The system of claim 4
wherein the first inter-controller interface circuit comprises a circuit adapted to couple with a switched fabric communication medium,
wherein a second inter-controller interface circuit of the second storage controller comprises a circuit adapted to couple with a switched fabric communication medium, and
wherein the system further comprises:
a switched fabric communication medium coupling the first inter-controller interface circuit with the second inter-controller interface circuit.
6. The system of claim 4
wherein the first inter-controller interface circuit comprises a serial attached SCSI (SAS) interface circuit.
7. The system of claim 6 further comprising:
a SAS switched fabric coupling the first inter-controller interface circuit with the second inter-controller interface circuit.
8. The system of claim 6
wherein the first storage controller is adapted to couple with the host system using one or more of: PowerPC processor local bus (PPC-PLB), peripheral component interconnect (PCI), and PCI Express.
9. The system of claim 4
wherein the DMA scatter-gather list comprises an entry that points to further scatter-gather list entries in the host system,
wherein the control circuit configures the interface circuits to transfer data in accordance with the scatter-gather list and in accordance with the further scatter-gather list entries.
10. A method for I/O request shipping between storage controllers, the method operable in a clustered storage system comprising a plurality of storage controllers, the method for shipping I/O requests between the storage controllers, the method comprising:
receiving in a first storage controller an I/O request from an attached host system, the request directed to a logical volume identified in the received I/O request, the request addressed to the first storage controller using a first target device address;
detecting that the logical volume is owned by a second storage controller wherein the second storage controller is identified by a second target device address;
generating a modified I/O request based on the received I/O request, wherein generation of the modified I/O request comprises replacing the first target device address with the second target device address; and
transmitting the modified I/O request to the second storage controller for processing therein.
11. The method of claim 10 further comprising:
receiving completion status within the first storage controller from the second storage controller indicating completion of the modified I/O request within the second storage controller; and
forwarding the completion status from the first storage controller to the host system.
12. The method of claim 10
wherein the modified I/O request comprises a direct memory access (DMA) scatter-gather list copied from the received I/O request, the DMA scatter-gather list indicating addresses in a memory of the host system for data associated with the received I/O request,
the method further comprising:
transferring data associated with the modified I/O request between the host system and the second storage controller through the first storage controller utilizing DMA capabilities of the first storage controller, wherein the transfer is in accordance with the DMA scatter-gather list.
13. The method of claim 12
wherein the step of transferring data further comprises:
configuring a DMA circuit of the first storage controller to transfer data between the second storage controller and the memory of the host system by DMA transfer based on the DMA scatter-gather list.
14. The method of claim 13
wherein the DMA scatter-gather list comprises an entry that points to further scatter-gather list entries in the host system,
wherein the step of configuring further comprises configuring the DMA circuit to transfer data in accordance with the scatter-gather list and in accordance with the further scatter-gather list entries.
15. A storage controller operable in a clustered storage system, the storage controller comprising:
ownership detection logic adapted to detect that an I/O request received from an attached host system is directed to a logical volume owned by another storage controller of the clustered storage system, wherein the storage controller is coupled with said other storage controller;
request shipping logic communicatively coupled with the ownership detection logic to receive the I/O request and adapted to generate a modified I/O request based on the I/O request, wherein the modified I/O request is a copy of the I/O request modified to address a target device address associated with said other storage controller; and
an inter-controller interface circuit communicatively coupled with the request shipping logic and adapted to couple the storage controller with said other storage controller, the inter-controller interface circuit having direct memory access (DMA) capability to access memory in the host system,
wherein the request shipping logic is adapted to transmit the modified I/O request to said other storage controller through the inter-controller interface circuit,
wherein the inter-controller interface circuit is adapted to transfer data associated with the modified I/O request between the memory of the host system and said other controller using the DMA capability.
16. The storage controller of claim 15
wherein the inter-controller interface circuit is adapted to receive a completion status from said other controller indicating completion of processing of the modified I/O request and is further adapted to return the completion status to the host system.
17. The storage controller of claim 15
wherein the request shipping logic is further adapted to configure the DMA capability of the inter-controller interface circuit to transfer the data between the memory of the host system and said other storage controller.
18. The storage controller of claim 17
wherein the I/O request comprises a DMA scatter-gather list, and
wherein the inter-controller interface circuit is further adapted to transfer the data in accordance with the DMA scatter-gather list.
19. The storage controller of claim 18
wherein the DMA scatter-gather list comprises an entry that points to further scatter-gather list entries in the host system,
wherein the inter-controller interface circuit is further adapted to transfer the data in accordance with the scatter-gather list and in accordance with the further scatter-gather list entries.