Scalable data access system and methods of eliminating controller bottlenecks
A data access system has host computers having front-end controllers nFE_SAN connected via a bus or network interconnect to back-end storage controllers nBE_SAN, and physical disk drives connected via network interconnect to the nBE_SANs to provide a distributed, high performance, policy based or dynamically reconfigurable, centrally managed, data storage acceleration system. The hardware and software architectural solutions eliminate BE_SAN controller bottlenecks and improve performance and scalability. In an embodiment, the nBE_SAN (BE_SAN) firmware recognize controller overload conditions, informs Distributed Resource Manager (DRM), and, based on the DRM provided optimal topology information, delegates part of its workload to additional controllers. The nFE_SAN firmware and additional hardware using functionally independent and redundant CPUs and memory that mitigate single points of failure and accelerates write performance. The nFE_SAN and FE_SAN controllers facilitate Converged I/O Interface by simultaneously supporting storage I/O and network traffic.
1. A method of writing write data to at least one storage device comprising:
generating a write request in a first process of a host computer;
passing the write request to an nFE_SAN driver in the host computer;
passing the write request from the nFE_SAN driver to a front-end storage controller (nFE SAN);
copying the write data to a cache memory of the nFE_SAN;
copying the write data to a cache memory of a secondary nFE_SAN (nFE_SAN-S),
generating a write lock request and transmitting the write lock request from the nFE_SAN over a network interconnect selected from a first and a second storage area interconnect to a back-end storage controller (nBE_SAN);
returning a write lock grant from the nBE_SAN to the nFE_SAN;
upon completing copying the write data to the cache memory of the nFE_SAN and the cache memory of the nFE_SAN-S and receiving the write lock grant from the nBE_SAN, the nFE_SAN driver providing a write complete signal to the first process of the host computer processor;
copying the write data over a network interconnect selected from the first and second storage area interconnect to the nBE_SAN;
writing, by the nBE_SAN, the write data to the at least one storage device;
the write complete signal being provided to the first process of the computer processor before the copying of the write data over the network interconnect from the nFE_SAN to the nBE_SAN completes;
where a cache memory selected from the cache memory of the nFE_SAN and the cache memory of the nBE_SAN further comprises a nonvolatile or flash memory.
2. The method of claim 1 wherein, upon failure of the cache memory of the nFE_SAN, the write data is copied from the cache memory of the nFE_SAN-S to the nBE_SAN.
3. A method of writing write data to at least one storage device comprising:
generating a write request in a host process of a computer;
passing the write request to a front-end storage controller (nFE_SAN);
copying the write data to a cache memory of an nFE_SAN;
copying the write data to a cache memory of a secondary nFE_SAN (nFE_SAN-S), generating a write lock request and transmitting the write lock request from the nFE_SAN over a network interconnect selected from a first and a second storage area interconnect to a selected back-end storage controller (SBE_SAN), the BBE SAN selected from a first and a second back-end storage controller (nBE_SAN) of a plurality of nBE_SANs according to logical block addresses (LBAs) identification associated with the write request, where the first nBE_SAN is assigned to write data associated with a first and second LBAs, and second nBE_SAN is assigned to write data associated with third LBAs, the first, second, and third LBAs being different and the first nBE_SAN being different from the second nBE_SAN;
returning a write lock grant from the SBE_SAN to the nFE_SAN;
upon completing copying the write data to the cache memory of the nFE_SAN and the cache memory of the nFE_SAN-S and receiving the write lock grant from the selected nBE_SAN, the nFE_SAN providing a write complete signal to the computer processor without waiting for copying the write data to the nBE_SAN to complete;
copying the write data over a network interconnect selected from the first and second storage area interconnect to the SBE_SAN; and
writing, by the selected SBE_SAN, the write data to the at least one storage device;
wherein, should the SBE_SAN be busy, an alternate nBE_SAN (ABE_SAN) is selected of the plurality of nBE_SANS, the ABE_SAN receiving the write data from the nFE_SAN into memory of the ABE_SAN, and later forwarding the write data to the SBE_SAN.
4. The method of claim 3 wherein, upon detecting overload of the first nBE_SAN by frequent accesses to the second LBAs, comprising copying data associated with the second LBAs from the first nBE_SAN to the second nBE_SAN and directing further references to the second LBAs to the second nBE_SAN and further references by the nFE_SAN are redirected to the second nBE_SAN.
5. A method of writing write data to at least one storage device comprising:
generating a write request in a first process of a host computer;
passing the write request to nFE_SAN driver in the host computer;
passing the write request from the nFE_SAN driver to a front-end storage controller (nFE_SAN);
copying the write data to a cache memory of the nFE_SAN;
copying the write data to a cache memory of a secondary nFE_SAN (nFE_SAN-S);
generating a write lock request and transmitting the write lock request from the nFE_SAN over a network interconnect selected from a first and a second storage area interconnect to a back-end storage controller (nBE_SAN);
returning a write lock grant from the nBE_SAN to the nFE_SAN;
upon completing copying the write data to the cache memory of the FE_SAN and the cache memory of the nFE_SAN-S and receiving the write lock grant from the nBE_SAN, the nFE_SAN driver providing a write complete signal to the first process of the host computer processor;
copying the write data over a network interconnect selected from the first and second storage area interconnect to the nBE_SAN;
writing, by the nBE_SAN, the write data to the at least one storage device;
the write complete signal being provided to the first process of the computer processor before the copying of the write data over the network interconnect from the nFE_SAN to the nBE_SAN completes;
where there are a plurality of nFE_SANs associated with the host computer.
6. The method of claim 3 wherein, should the ABE_SAN become overloaded, the system automatically reconfigures to use a different nBE_SAN as an alternate ABE_SAN (AABE_SAN) in addition to the ABE_SAN.
7. A method of writing write data to at least one storage device comprising:
generating a write request in a first process of a host computer;
passing the write request to an nFE_SAN driver in the host computer;
passing the write request from the nFE_SAN driver to a front-end storage controller (nFE SAN);
copying the write data to a cache memory of the nFE_SAN;
copying the write data to a cache memory of a secondary nFE_SAN (nFE_SAN-S), the nFE_SAN and nFE_SAN-S maintaining two copies of the write data;
generating a write lock request and transmitting the write lock request from the nFE_SAN over a network interconnect selected from a first and a second storage area interconnect to a back-end storage controller (nBE_SAN);
returning a write lock grant from the nBE_SAN to the nFE_SAN;
upon completing copying the write data to the cache memory of the nFE_SAN and the cache memory of the nFE_SAN-S and receiving the write lock grant from the nBE_SAN, the nFE_SAN driver providing a write complete signal to the first process of the host computer processor;
copying the write data over a network interconnect selected from the first and second storage area interconnect from the nFE_SAN to the nBE_SAN, and after the write data is copied from the nFE_SAN to the nBE_SAN, releasing one copy of the write data from memory of either the nFE_SAN or the nFE_SAN-S;
writing, by the nBE_SAN, the write data to the at least one storage device;
the write complete signal being provided to the first process of the computer processor before the copying of the write data over the network interconnect from the nFE_SAN to the nBE_SAN completes;
where a cache memory selected from the cache memory of the nFE_SAN and the cache memory of the nBE_SAN further comprises a nonvolatile or flash memory.