Systems and methods for shipping an I/O operation to prevent replication failure
View Patent ↗A computer-implemented method for shipping I/O operations to prevent replication failure may include 1) attempting to perform an I/O operation in a system configured to replicate data from a data cluster to another data cluster, 2) detecting a failure in at least part of the attempt to perform the I/O operation that threatens to fail the system's replication of data from the data cluster to the other data cluster, and, in response to detecting the failure, 3) shipping the I/O operation from a node originally responsible for servicing the I/O operation to another node to complete the I/O operation without failing the system's replication of data from the data cluster to the other data cluster. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for shipping I/O operations to prevent replication failure, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
attempting to perform an I/O operation in a system configured to replicate data from a data cluster to another data cluster;
detecting a failure in at least part of the attempt to perform the I/O operation that threatens to fail the system's replication of data from the data cluster to the other data cluster;
prior to shipping the I/O operation:
determining a performance impact of shipping the I/O operation from a node originally responsible for servicing the I/O operation to another node within the same data cluster;
determining that the performance impact of shipping the I/O operation from the node originally responsible for servicing the I/O operation to the other node within the same data cluster satisfies at least one performance-related criterion;
shipping the I/O operation, in response to detecting the failure in at least part of the attempt to perform the I/O operation that threatens to fail the system's replication of data from the data cluster to the other data cluster, from the node originally responsible for servicing the I/O operation to the other node within the same data cluster to complete the I/O operation without failing the system's replication of data from the data cluster to the other data cluster, wherein:
the performance-related criterion includes at least one of a latency threshold, a CPU usage threshold, and a bandwidth threshold.
2. The computer-implemented method according to claim 1 ,
further comprising determining, at a later point in time, that the performance impact of shipping the I/O operation from the node originally responsible for servicing the I/O operation to the other node within the same data cluster fails to satisfy the performance-related criterion.
3. The computer-implemented method according to claim 2 , further comprising
gracefully stopping, on a node where the failure occurred, an application that caused the attempt to perform the I/O operation.
4. The computer-implemented method according to claim 1 , wherein shipping the I/O operation comprises:
determining that the I/O operation comprises a replication I/O operation associated with the system's replication of data from the data cluster to the other data cluster, as opposed to an application I/O operation associated with servicing an application;
shipping the I/O operation in response to the determination.
5. The computer-implemented method according to claim 1 , wherein the node originally responsible for servicing the I/O operation comprises a node within the other data cluster.
6. The computer-implemented method according to claim 5 , wherein shipping the I/O operation comprises:
determining that the failure is at least one of non-permanent and non-global;
shipping the I/O operation in response to the determination.
7. The computer-implemented method according to claim 1 , wherein the detection of the failure occurs during at least one of:
an initial synchronization operation that automatically performs synchronization to ensure that the other data cluster is up to date with the data cluster;
a full synchronization operation that transfers all data between the data cluster and the other data cluster;
a differential synchronization operation that transfers only differing data between the data cluster and the other data cluster.
8. The computer-implemented method according to claim 1 , wherein the system implements a replication log.
9. The computer-implemented method according to claim 8 , wherein the replication log comprises a bunker replication log for bunker-based replication in which the data cluster and the other data cluster are physically isolated from each other.
10. The computer-implemented method according to claim 8 , wherein the replication log, and a receiving data volume that receives the attempt to perform the I/O operation, are stored on separate storage devices.
11. The computer-implemented method according to claim 10 , wherein the failure comprises a failure to access the receiving data volume.
12. The computer-implemented method according to claim 11 , wherein:
the node originally responsible for servicing the I/O operation comprises a node within the data cluster;
the failure comprises a failure to write to the receiving data volume after the write has been recorded in the replication log;
shipping the I/O operation comprises shipping an application I/O operation associated with servicing an application;
the method further comprises shipping the write preserved in the replication log to the other node within the same data cluster.
13. The computer-implemented method according to claim 8 , wherein the failure comprises a failure to access the replication log.
14. The computer-implemented method according to claim 13 , wherein:
the I/O operation comprises an I/O operation on the replication log;
shipping the I/O operation comprises shipping the I/O operation so that the I/O operation is performed on the replication log by the other node within the same data cluster.
15. The computer-implemented method according to claim 13 , further comprising, in response to detecting the failure, switching a log maintenance role associated with the replication log from a current node to a different node.
16. The computer-implemented method according to claim 1 , wherein detecting the failure comprises detecting the failure during a recovery operation that attempts to recover replicated data in response to a previous failure.
17. The computer-implemented method according to claim 1 , wherein the I/O operation comprises an I/O operation on a data change map, wherein the data change map indicates where data differs between the data cluster and the other data cluster.
18. A system for shipping I/O operations to prevent replication failure, the system comprising:
an I/O module programmed to attempt to perform an I/O operation in the system which is configured to replicate data from a data cluster to another data cluster;
a detection module programmed to detect a failure in at least part of the attempt to perform the I/O operation that threatens to fail the system's replication of data from the data cluster to the other data cluster;
a shipping module programmed to:
prior to shipping the I/O operation:
determine a performance impact of shipping the I/O operation from a node originally responsible for servicing the I/O operation to another node within the same data cluster;
determine that the performance impact of shipping the I/O operation from the node originally responsible for servicing the I/O operation to the other node within the same data cluster satisfies at least one performance-related criterion;
ship the I/O operation, in response to detecting the failure in at least part of the attempt to perform the I/O operation that threatens to fail the system's replication of data from the data cluster to the other data cluster, from the node originally responsible for servicing the I/O operation to the other node within the same data cluster to complete the I/O operation without failing the system's replication of data from the data cluster to the other data cluster;
at least one processor configured to execute the I/O module, the detection module, and the shipping module, wherein:
the performance-related criterion includes at least one of a latency threshold, a CPU usage threshold, and a bandwidth threshold.
19. The system according to claim 18 , wherein the node originally responsible for servicing the I/O operation comprises a node within the other data cluster.
20. A non-transitory computer-readable-storage medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
attempt to perform an I/O operation in a system configured to replicate data from a data cluster to another data cluster;
detect a failure in at least part of the attempt to perform the I/O operation that threatens to fail the system's replication of data from the data cluster to the other data cluster;
prior to shipping the I/O operation:
determine a performance impact of shipping the I/O operation from a node originally responsible for servicing the I/O operation to another node within the same data cluster;
determine that the performance impact of shipping the I/O operation from the node originally responsible for servicing the I/O operation to the other node within the same data cluster satisfies at least one performance-related criterion;
ship the I/O operation, in response to detecting the failure in at least part of the attempt to perform the I/O operation that threatens to fail the system's replication of data from the data cluster to the other data cluster, from the node originally responsible for servicing the I/O operation to the other node within the same data cluster to complete the I/O operation without failing the system's replication of data from the data cluster to the other data cluster, wherein:
the performance-related criterion includes at least one of a latency threshold, a CPU usage threshold, and a bandwidth threshold.