Systems and methods for obtaining ultra-high data availability and geographic disaster tolerance
Network data storage systems and methods allow computers reading and writing data at a plurality of data centers separated by, potentially, large distances to replicate data between sites such that the data is protected from failures, including complete Site failures, while not allowing network latency to significantly impede the performance of read or write operations. Continued access to all data is provided even after a single failure of any component of the system or after any complete failure of all equipment located at any single geographic region or any failure that isolates access to any single geographic region. Write data is replicated synchronously from Active Sites, e.g., sites where servers are writing data to storage resources, to Protection Sites located sufficiently close to Active Sites such that network latency will not significantly impact performance, but sufficiently far apart such that a regional disaster is unlikely to affect both sites. Write data is then asynchronously copied to other sites, potentially including one or more Active sites, located at greater distances.
1. A method of providing data availability and fault tolerance in a data storage network having a first protection domain comprising a first site and a second site, and a second protection domain separate from the first protection domain, the second protection domain having a third site, each site including at least one control node, the method comprising the steps of:
storing a write request received from a host system to a first cache, the first cache corresponding to a first node in the first site;
transmitting the write request to a second node at the second site;
storing the write request received from the first node to a second cache in the second node, the write request in each of the first and second caches being directed to a first leg of a multiple-leg mirror volume distributed across the first and second protection domains, the write request in the second cache being a protection write request for temporary storage until the write request from the host is safely stored on at least two legs of the multiple-leg mirror volume;
receiving at the first node an acknowledgement from the second node that the write request was received by the second node and stored in the second cache without also being written by the second node to a physical storage device;
in response to receiving the acknowledgment from the second node, acknowledging to the host system by the first node that the write request is complete;
sending the write request to a third node at a third site within the second protection domain, the write request to the third node being directed to a second leg of the multiple-leg mirror volume; and
upon the write request to the third node being stored on a physical storage device in the second protection domain, deleting the protection write request in the second cache without storing the protection write request on a physical storage device in the first protection domain.
2. The method of claim 1 , further comprising:
sending the write request to another node in the first site simultaneously with transmitting the write request to the second node in the second site; and
storing the write request received from the first node to a cache in said another node; and acknowledging receipt of the write request by said another node to the first node.
3. The method of claim 1 , further comprising:
sending the write request from the second node to another node in the second site;
storing the write request received from the second node to a cache in said another node; and
acknowledging receipt of the write request by said another node to the second node.
4. The method of claim 1 , wherein the first site is located within about 80 kilometers of the second site.
5. The method of claim 1 , wherein sites within the first protection domain are located greater than about 80 kilometers from sites within the second protection domain.
6. The method of claim 1 , wherein the nodes in the first site are interconnected with the nodes of the second site over a WAN interconnect.
7. The method of claim 1 , wherein the nodes in the first site are interconnected over one of a LAN interconnect, a MAN interconnect, a bus, an Fibre channel interconnect a SCSI interconnect, and an Infiniband interconnect.
8. The method of claim 1 , further comprising:
determining by the first node whether any other nodes have data in cache corresponding to a data range associated with the write request; and
if so, sending a cache invalidate message to said any other nodes identifying the data range.
9. The method of claim 1 , further comprising:
writing data identified in the write request to physical storage.
10. The method of claim 9 , wherein writing is performed by all nodes maintaining physical copies of a data range identified by the write request.
11. A data storage network that provides high data availability and fault tolerance, the network comprising:
a first protection domain including a first site having a first control node and a second site having a second control node, wherein the first and second control nodes each have a cache; and
a second protection domain having third site having a third control node, the third control node having a cache;
wherein the first control node is configured to:
i) store a write request received from a host within the first site to its cache;
ii) send the write request to the second node for storing in its cache, the write request in each of the first and second caches being directed to a first leg of a multiple-leg mirror volume distributed across the first and second protection domains, the write request in the second cache being a protection write request for temporary storage until the write request from the host is safely stored on at least two legs of the multiple-leg mirror volume; and
iii) upon receiving an acknowledgement from the second node that the write request is stored in its cache, the acknowledgment being sent upon the write request being stored in the second cache without also being written by the second node to a physical storage device:
a) acknowledge to the host that the write request is complete;
b) send the write request to the third control node, the write request to the third control node being directed to a second leg of the multiple-leg mirror volume; and
c) upon the write request to the third control node being stored on a physical storage device in the second protection domain, delete the protection write request in the second cache without storing the protection write request on a physical storage device in the first protection domain.
12. The data storage network of claim 11 , wherein the first control node is further configured to:
iv) determine whether any other control nodes have data in cache corresponding to a data range associated with the write request and, if so, send a cache invalidate message to said any other control nodes.
13. The data storage network of claim 11 , wherein the first control node is further configured to:
iv) send the write request to another control node in the first site simultaneously with sending the write request to the second node in the second site.
14. The data storage network of claim 11 , wherein the first protection domain is located greater than about 80 kilometers from the second protection domain.
15. The data storage network of claim 11 , wherein the first site is located within about 80 kilometers of the second site.
16. The data storage network of claim 11 , wherein the first site includes one or more physical storage resources, and wherein the first control node is further configured to: iv) access and/or modify data identified in the write request.
17. A data storage network control node for use as a first control node in a data storage network, comprising:
a cache; and
a processor that implements logic that is configured to:
i) store a write request received from a host system in the cache, the host system and first control node being in a first site;
ii) send the write request to a second control node in a second site, said first and second sites being part of a first protection domain, the write request being sent for storing in a cache of the second control node, the write request in the caches of the first and second control nodes being directed to a first leg of a multiple-leg mirror volume distributed across the first protection domain and a second protection domain separate from the first protection domain, the write request in the cache of the second control node being a protection write request for temporary storage until the write request from the host is safely stored on at least two legs of the multiple-leg mirror volume; and
iii) upon receiving an acknowledgement from the second control node that the write request is stored in its cache, the acknowledgment being sent upon the write request being stored in the second cache without also being written by the second node to a physical storage device:
a) acknowledge to the host system that the write request is complete;
b) send the write request to a third control node in the second protection domain, the write request to the third control node being directed to a second leg of the multiple-leg mirror volume; and
c) upon the write request to the third control node being stored on a physical storage device in the second protection domain, delete the protection write request in the second cache without storing the protection write request on a physical storage device in the first protection domain.
18. The data storage network control node of claim 17 , wherein the logic is further configured to:
iv) determine whether any other control nodes have data in cache corresponding to a data range associated with the write request and, if so, send a cache invalidate message to said any other control nodes.
19. The data storage network control node of claim 17 , wherein the logic is further configured to:
iv) send the write request to another control node in the first site simultaneously with sending the write request to the second node in the second site.
20. The data storage network control node of claim 17 , wherein the first protection domain is located greater than about 80 kilometers from the second protection domain.
21. The data storage network control node of claim 17 , wherein the first site is located within about 80 kilometers of the second site.
22. The data storage network control node of claim 17 , wherein the first site includes one or more physical storage resources, and wherein the logic is further configured to:
iv) access and/or modify data identified in the write request.
23. The network of claim 11 , wherein if the first site and/or the first control node fails, a data range identified by the write request is accessible to host systems via one or more of the second control node or another control node in the second site or a control node in the third site in the second protection domain, said third site including the third control node.
24. The network of claim 11 , wherein if the first site and/or physical storage in the first site fails, a data range identified by the write request is recoverable using change logs maintained by one or more of the second control node, another control node in the second site or a control node in the third site in the second protection domain, said third site including the third control node.
25. The network of claim 11 , if the first site and/or the first control node fails, the second control node or another control node in the second site is available to the host and other host systems to process write requests for at least the data range identified by the write request.
26. The data storage network control node of claim 17 , wherein functions of the logic are performed in a normal operating mode in which none of the nodes, sites and protection domains have failed, and wherein the logic is further configured to:
1) upon failure of one of the nodes, (a) initially suspend input/output operations at all nodes in the same site as the failed node, (b) prevent the node from receiving future write requests, (c) identify protection copies, stored on other nodes, of primary data blocks stored on the failed node, and upgrading the protection copies to become primary copies, and (d) thereafter resume input/output operations without participation of the failed node;
(2) upon failure of one of the sites, (a) initially suspend input/output operations at all sites, (b) prevent the failed site from receiving future write requests, (c) identify protection copies, stored in other sites, of primary data blocks stored at the failed site, and upgrade the protection copies to become primary copies, and (d) thereafter resume input/output operations without participation of the failed site; and
(3) upon failure of one of the protection domains, (a) initially suspend input/output operations to all volumes for which there may be data lost due to failure of the protection domain, (b) back up a write-order-fidelity state of the volumes to earlier versions prior to the writing of the lost data, and (c) resume input/output operations to the volumes and restarting any applications using the volumes.
27. The data storage network control node of claim 26 , wherein the failed site is a protection site for an active site, and wherein the logic is further configured to select from among a set of options for the continued operation, the set of options including operating indefinitely with reduced resiliency; failing the active site; entering a write-through operating mode with respect to write requests sent to the second protection domain; and operating with reduced resiliency only while migrating applications off the active site.