Systems and methods for shipping I/O operations to prevent replication failure
View Patent ↗A computer-implemented method for shipping I/O operations to prevent replication failure may include 1) attempting to perform an I/O operation in a system configured to replicate data from a data cluster to another data cluster, 2) detecting a failure in at least part of the attempt to perform the I/O operation that threatens to fail the system's replication of data from the data cluster to the other data cluster, and, in response to detecting the failure, 3) shipping the I/O operation from a node originally responsible for servicing the I/O operation to another node to complete the I/O operation without failing the system's replication of data from the data cluster to the other data cluster. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for shipping I/O operations to prevent replication failure, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
attempting to perform an I/O operation in a replication system configured to replicate data from a data cluster to another data cluster, the replication system comprising a log maintenance node that maintains a replication log for replication;
detecting a failure in at least part of the attempt to perform the I/O operation that threatens to fail the replication system's replication of data from the data cluster to the other data cluster, the replication system being configured in a synchronous mode that attempts to perform arriving I/O operations through the log maintenance node;
determining that the I/O operation constitutes a replication I/O operation associated with the replication system's replication of data from the data cluster to the other data cluster as opposed to an application I/O operation associated with servicing an application, performance-related criteria indicating that application I/O is more time sensitive than replication I/O;
shipping the I/O operation, based at least in part on the detecting the failure in at least part of the attempt to perform the I/O operation that threatens to fail the replication system's replication of data from the data cluster to the other data cluster, from the log maintenance node to another node within the same data cluster to complete the I/O operation without failing the replication system's replication of data from the data cluster to the other data cluster.
2. The computer-implemented method according to claim 1 , further comprising, prior to shipping the I/O operation:
determining a performance impact of shipping the I/O operation from the log maintenance node to the other node;
determining that the performance impact of shipping the I/O operation from the log maintenance node to the other node satisfies the performance-related criteria.
3. The computer-implemented method according to claim 1 , further comprising:
detecting a failure in at least part of an attempt to perform another I/O operation that threatens to fail the replication system's replication of data from the data cluster to the other data cluster;
determining that a performance impact of shipping the other I/O operation from the log maintenance node to another node fails to satisfy the performance-related criteria.
4. The computer-implemented method according to claim 3 , further comprising gracefully stopping, on the node where the failure of the attempt to perform the other I/O operation occurred, the application that caused the attempt to perform the other I/O operation.
5. The computer-implemented method according to claim 1 , wherein the log maintenance node ships the I/O operation to a receiving node in the data cluster that originally received the I/O operation.
6. The computer-implemented method according to claim 5 ,
wherein, prior to the log maintenance node shipping the I/O operation to the receiving node, the receiving node shipped the I/O operation to the log maintenance node.
7. The computer-implemented method according to claim 1 ,
wherein:
the other node already possesses data for writing the I/O operation;
the I/O operation is shipped by simply instructing the other node to write the data that the other node already possesses.
8. The computer-implemented method according to claim 1 , wherein the log maintenance node ships the I/O operation to a node in the data cluster that is different than a receiving node that originally received the I/O operation.
9. The computer-implemented method according to claim 1 , wherein the replication log comprises a bunker replication log for bunker-based replication in which the data cluster and the other data cluster are physically isolated from each other.
10. The computer-implemented method according to claim 1 , wherein the replication log, and a receiving data volume that receives the attempt to perform the I/O operation, are stored on separate storage devices.
11. The computer-implemented method according to claim 10 , wherein the failure comprises a failure to access the receiving data volume.
12. The computer-implemented method according to claim 7 , wherein the other node comprises a receiving node in the data cluster that originally received the I/O operation.
13. The computer-implemented method according to claim 1 , wherein the failure comprises a failure to access the replication log.
14. The computer-implemented method according to claim 13 , wherein:
the I/O operation comprises an I/O operation on the replication log;
shipping the I/O operation comprises shipping the I/O operation so that the I/O operation is performed on the replication log by the other node.
15. The computer-implemented method according to claim 13 , further comprising, in response to detecting the failure, switching a log maintenance role associated with the replication log from the log maintenance node to a different node.
16. The computer-implemented method according to claim 1 , wherein detecting the failure in at least part of the attempt to perform the I/O operation triggers at least a partial deviation from the synchronous mode.
17. The computer-implemented method according to claim 1 , wherein:
the I/O operation is directed to the replication log;
the I/O operation is shipped at least in part by transferring log metadata to a receiving node in the data cluster that originally received the I/O operation.
18. A system for shipping I/O operations to prevent replication failure, the system comprising:
an I/O module programmed to attempt to perform an I/O operation in a replication system configured to replicate data from a data cluster to another data cluster, the replication system comprising a log maintenance node that maintains a replication log for replication;
a detection module programmed to detect a failure in at least part of the attempt to perform the I/O operation that threatens to fail the replication system's replication of data from the data cluster to the other data cluster, the replication system being configured in a synchronous mode that attempts to perform arriving I/O operations through the log maintenance node;
a shipping module programmed to:
determine that the I/O operation constitutes a replication I/O operation associated with the replication system's replication of data from the data cluster to the other data cluster as opposed to an application I/O operation associated with servicing an application, performance-related criteria indicating that application I/O is more time sensitive than replication I/O;
ship the I/O operation, based at least in part on the detecting the failure in at least part of the attempt to perform the I/O operation that threatens to fail the replication system's replication of data from the data cluster to the other data cluster, from the log maintenance node to another node within the same data cluster to complete the I/O operation without failing the replication system's replication of data from the data cluster to the other data cluster;
at least one processor configured to execute the I/O module, the detection module, and the shipping module.
19. The system according to claim 18 , wherein the shipping module is programmed to ship the I/O operation to a receiving node in the data cluster that originally received the I/O operation.
20. A non-transitory computer-readable-storage medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
attempt to perform an I/O operation in a replication system configured to replicate data from a data cluster to another data cluster, the replication system comprising a log maintenance node that maintains a replication log for replication;
detect a failure in at least part of the attempt to perform the I/O operation that threatens to fail the replication system's replication of data from the data cluster to the other data cluster, the replication system being configured in a synchronous mode that attempts to perform arriving I/O operations through the log maintenance node;
determine that the I/O operation constitutes a replication I/O operation associated with the replication system's replication of data from the data cluster to the other data cluster as opposed to an application I/O operation associated with servicing an application, performance-related criteria indicating that application I/O is more time sensitive than replication I/O;
ship the I/O operation, based at least in part on the detecting the failure in at least part of the attempt to perform the I/O operation that threatens to fail the replication system's replication of data from the data cluster to the other data cluster, from the log maintenance node to another node in the same data cluster to complete the I/O operation without failing the replication system's replication of data from the data cluster to the other data cluster.