Scalable data storage architecture and methods of eliminating I/O traffic bottlenecks
View Patent ↗A Storage Area Network (SAN) system has host computers, front-end SAN controllers (FE_SAN) connected via a bus or network interconnect to back-end SAN controllers (BE_SAN), and physical disk drives connected via network interconnect to the BE_SANs to provide distributed high performance centrally managed storage. Described are hardware and software architectural solutions designed to eliminate I/O traffic bottlenecks, improve scalability, and reduce the overall cost of SAN systems. In an embodiment, the BE_SAN has firmware to recognize when, in order to support a multidisc volume, such as a RAID volume, it is configured to support, it requires access to a physical disk attached to a second BE_SAN; when such a reference is recognized it passes assess commands to the second BE_SAN. Buffer memory of each FE_SAN is mapped into application memory space to increase access speed, where multiple hosts share an LBA the BE_SAN tracks writes and invalidates the unwritten buffers.
1. A data processing system comprising:
a plurality of host CPUs;
at least a first and a second Front End SAN controller (FE_SAN), where each FE_SAN is coupled to at least one host CPU of the plurality of host CPUs;
at least a first Back End SAN controller (BE_SAN), each BE_SAN being coupled via at least one back-end network to a plurality of storage devices, each BE_SAN associated with a plurality of logical block addresses (LBAs);
a storage area network fabric interconnecting the FE_SANs with the BE_SAN; and
FE_SAN virtualization firmware configured to control the FE_SAN controllers and BE_SAN virtualization firmware configured to execute on BE_SAN controllers;
wherein the FE_SAN virtualization firmware and the BE_SAN virtualization firmware are together configured to read data from the first BE_SAN into a first memory-mapped buffer of the first FE_SAN when data of a first LBA is required by a host CPU coupled to the first FE_SAN;
wherein the FE_SAN virtualization firmware is configured such that, when a host CPU modifies data in the first memory-mapped buffer, the FE_SAN copies data from the first memory-mapped buffer to at least one corresponding buffer allocated on the BE_SAN associated with the first LBA on write commands; and
wherein the FE_SAN virtualization firmware is configured such that, if the second FE_SAN controller maps physical memory as a second buffer of the first LBA, and if data of the first buffer is modified by a host coupled to the first FE_SAN, the second FE_SAN is notified to invalidate data of the second buffer;
wherein the back-end network comprises a back-end network fabric and the at least one BE_SAN has firmware for using a storage control protocol on the back-end network fabric.
2. The system of claim 1 wherein the BE_SAN firmware for using a storage control protocol on the back-end network fabric is configured to use SCSI over Ethernet.
3. A data processing system comprising:
a plurality of host CPUs;
at least a first and a second Front End SAN controller (FE_SAN), where each FE_SAN is coupled to at least one host CPU of the plurality of host CPUs;
at least a first Back End SAN controller (BE_SAN), each BE_SAN being coupled via at least one back-end network to a plurality of storage devices, each BE_SAN associated with a plurality of logical block addresses (LBAs);
a storage area network fabric interconnecting the FE_SANs with the BE_SAN; and
FE_SAN virtualization firmware configured to control the FE_SAN controllers and BE_SAN virtualization firmware configured to execute on BE_SAN controllers;
wherein the FE_SAN virtualization firmware and the BE_SAN virtualization firmware are together configured to read data from the first BE_SAN into a first memory-mapped buffer of the first FE_SAN when data of a first LBA is required by a host CPU coupled to the first FE_SAN;
wherein the FE_SAN virtualization firmware is configured such that, when a host CPU modifies data in the first memory-mapped buffer, the FE_SAN copies data from the first memory-mapped buffer to at least one corresponding buffer allocated on the BE_SAN associated with the first LBA on write commands; and
wherein the FE_SAN virtualization firmware is configured such that, if the second FE_SAN controller maps physical memory as a second buffer of the first LBA, and if data of the first buffer is modified by a host coupled to the first FE_SAN, the second FE_SAN is notified to invalidate data of the second buffer;
wherein there is a second BE_SAN controller coupled to the FE_SANs over the storage area network fabric;
wherein a second LBA is associated with the second BE_SAN, and wherein the FE_SAN virtualization firmware is configured such that, when the first FE_SAN maps a portion of physical memory as a first buffer of the second LBA and data of the second buffer in modified, data of the second buffer is copied to a corresponding buffer allocated on the second BE_SAN on write commands; and
wherein the FE_SAN virtualization firmware is configured such that, if the second FE_SAN controller maps physical memory as a second buffer of the second LBA and if data of the first buffer of the second LBA is modified, the second FE_SAN is notified to invalidate data of the second buffer of the second LBA.
4. The system of claim 1 wherein the FE_SAN and BE_SAN firmware is configured for writeback caching with mirroring.
5. The system of claim 1 wherein the FE_SAN and BE_SAN firmware is configured for writeback caching without mirroring.
6. The system of claim 1 wherein the FE_SAN and BE_SAN firmware is configured to store data organized as a “redundant array of independent disks” (RAID) dataset across multiple storage devices.
7. The system of claim 1 wherein the FE_SAN and BE_SAN firmware is configured to store data of a RAID dataset across multiple storage devices coupled to at least two BE_SANs a hot-spare storage device substituted for a failed storage devices in RAID-organized virtual disks may be coupled to a different BE_SAN than the failed storage device.
8. A method of operating storage on a data processing system having a plurality of host CPUs comprising:
providing at least a first and a second Front End SAN controller (FE_SAN), where each FE_SAN is coupled to at least one different host CPU of the host CPUs;
providing at least a first Back End SAN controller (BE_SAN), each BE_SAN being coupled via at least one back-end network to a plurality of storage devices, each BE_SAN associated with a plurality of logical block addresses (LBAs);
coupling FE_SANs to the BE_SAN with a storage area network fabric;
reading data from a first LBA from the BE_SAN to a first memory-mapped buffer associated with the first FE_SAN;
allowing an application to directly access the first memory-mapped buffer;
copying data from the first buffer to at least one corresponding buffer allocated on a BE_SAN associated with the first LBA;
wherein the FE_SAN virtualization firmware is configured such that, if a second FE_SAN controller maps physical memory as a second buffer of the first LBA while the first buffer of the first LBA is in use, and if data of the first buffer is modified by a host coupled to the first FE_SAN, the second FE_SAN is notified to invalidate data of the second buffer;
wherein the back-end network comprises a back-end network fabric and the at least one BE_SAN has firmware for using a storage control protocol on the back-end network fabric.
9. The method of claim 8 wherein the BE_SAN firmware for using a storage control protocol on the back-end network fabric is configured to use SCSI over Ethernet.
10. The method of claim 8 wherein the FE_SAN and BE_SAN firmware is configured for writeback caching with mirroring.
11. The method of claim 8 wherein the FE_SAN and BE_SAN firmware is configured for writeback caching without mirroring.
12. The method of claim 8 wherein the FE_SAN and BE_SAN firmware is configured to store data organized as a “redundant array of independent disks” (RAID) dataset across multiple storage devices.
13. The method of claim 8 wherein the FE_SAN and BE_SAN firmware is configured to store data of a RAID dataset across multiple storage devices coupled to at least two BE_SANs a hot-spare storage device substituted for a failed storage devices in RAID-organized virtual disks may be coupled to a different BE_SAN than the failed storage device.