Highly resilient synchronous replication with automatic recovery
In one aspect, automatic recovery of a synchronous replication session in response to an error is provided for a storage system that includes a source and target sites. During an active sync replication session in which a state machine indicates the system is operating in sync, an aspect includes monitoring input/output (IO) operations. Upon determining an occurrence of the error in which data has been persisted at the source site but not at the target site, an aspect includes discontinuing replication to the target site and transitioning the state machine from a sync state to a tripped state. Upon determining, during the tripped state, resources exist to conduct sync replication remote data transfer operations, transition the state machine to an async_to_sync state. The async_to_sync state causes the storage system to initiate a recovery operation to return the source and target sites to the sync state.
1. A method to perform automatic recovery of a synchronous (sync) replication session in response to an error condition for a storage system including a source site and a target site, the method comprising:
during an active sync replication session between the source site and the target site in which a state machine at the storage system indicates the system is operating in a sync state, monitoring input/output (IO) operations, wherein the active sync replication session includes data replication from the source site to the target site, and wherein the sync state indicates the source site and the target site are operating in sync; and
upon determining, during the active sync replication session, an occurrence of the error condition in which data subject for updating at the target site has been persisted at the source site and not yet persisted at the target site:
discontinuing IO replication to the target site; and
transitioning the state machine from the sync state to a tripped state; and
upon determining, during the tripped state, resources exist to conduct sync replication remote data transfer operations, transition the state machine to an async_to_sync state, the async_to_sync state causing the storage system to initiate a recovery operation, the recovery operation configured to return the source site and the target site to the sync state.
2. The method of claim 1 , wherein during the async_to_sync state, the recovery operation comprises:
locating a most recent recovery snap set pair;
initiating an async-to-sync replication cycle using the recovery snap set pair, the async-to-sync replication cycle resulting in sync replication data transfer; and
upon completion of the sync replication data transfer, transitioning the state machine to a send_cycle state.
3. The method of claim 2 , further comprising:
upon determining an occurrence of an IO failure during the async-to-sync replication cycle, transitioning the state machine back to the tripped state.
4. The method of claim 2 , wherein during the send_cycle state, the method further comprises:
performing async-to-sync transition cycle transfer concurrently with sync replication;
monitoring progress of the async-to-sync transition cycle transfer and sync replication; and
upon completion of the async-to-sync transition cycle transfer and the sync replication, transitioning the state machine back to the sync state.
5. The method of claim 4 , wherein upon determining an occurrence of an IO failure during the async-to-sync transition cycle transfer or the sync replication, transitioning the state machine back to the tripped state.
6. The method of claim 1 , wherein the error condition comprises an IO error condition in which a set timeout value is 30 seconds and the data subject to updating at the target site has not been persisted prior to expiration of the timeout value.
7. The method of claim 1 , wherein a plurality of state machines is provided in the storage system, and each of the state machines corresponds to one of a plurality of synchronous replication sessions.
8. A system for performing automatic recovery of a synchronous (sync) replication session in response to an error condition for a storage system including a source site and a target site, the system comprising:
a memory comprising computer-executable instructions; and
a processor operable by a storage system, the processor executing the computer-executable instructions, the computer-executable instructions when executed by the processor cause the processor to perform operations comprising:
during an active sync replication session between the source site and the target site in which a state machine at the storage system indicates the system is operating in a sync state, monitoring input/output (IO) operations, wherein the active sync replication session includes data replication from the source site to the target site, and wherein the sync state indicates the source site and the target site are operating in sync; and
upon determining, during the active sync replication session, an occurrence of the error condition in which data subject for updating at the target site has been persisted at the source site and not yet persisted at the target site:
discontinuing replication to the target site; and
transitioning the state machine from the sync state to a tripped state; and
upon determining, during tripped state, resources exist to conduct sync replication remote data transfer operations, transition the state machine to an async_to_sync state, the async_to_sync state causing the storage system to initiate a recovery operation, the recovery operation configured to return the source site and the target site to the sync state.
9. The system of claim 8 , wherein during the async_to_sync state, the recovery operation comprises:
locating a most recent recovery snap set pair;
initiating an async-to-sync replication cycle using the recovery snap set pair, the async-to-sync replication cycle resulting in sync replication data transfer; and
upon completion of the sync replication data transfer, transitioning the state machine to a send_cycle state.
10. The system of claim 9 , wherein the operations further include:
upon determining an occurrence of an IO failure during the async-to-sync replication cycle, transitioning the state machine back to the tripped state.
11. The system of claim 9 , wherein during the send_cycle state, the operations further comprise:
performing async-to-sync transition cycle transfer concurrently with sync replication;
monitoring progress of the async-to-sync transition cycle transfer and sync replication; and
upon completion of the async-to-sync transition cycle transfer and the sync replication, transitioning the state machine back to the sync state.
12. The system of claim 11 , wherein upon determining an occurrence of an IO failure during the async-to-sync transition cycle transfer or the sync replication, transitioning the state machine back to the tripped state.
13. The system of claim 8 , wherein the error condition comprises an IO error condition in which a set timeout value is 30 seconds and the data subject to updating at the target site has not been persisted prior to expiration of the timeout value.
14. The system of claim 8 , wherein storage system includes a plurality of state machines, and each of the state machines corresponds to one of a plurality of synchronous replication sessions.
15. A computer program product embodied on a non-transitory computer readable medium, the computer program product including instructions that, when executed by a computer, causes the computer to perform operations comprising:
during an active sync replication session between the source site and the target site in which a state machine at the storage system indicates the system is operating in a sync state, monitoring input/output (IO) operations, wherein the active sync replication session includes data replication from the source site to the target site, and wherein the sync state indicates the source site and the target site are operating in sync; and
upon determining, during the active sync replication session, an occurrence of the error condition in which data subject for updating at the target site has been persisted at the source site and not yet persisted at the target site:
discontinuing IO replication to the target site; and
transitioning the state machine from the sync state to a tripped state; and
upon determining, during the tripped state, resources exist to conduct sync replication remote data transfer operations, transition the state machine to an async_to_sync state, the async_to_sync state causing the storage system to initiate a recovery operation, the recovery operation configured to return the source site and the target site to the sync state.
16. The computer program product of claim 15 , wherein during the async_to_sync state, the recovery operation comprises:
locating a most recent recovery snap set pair;
initiating an async-to-sync replication cycle using the recovery snap set pair, the async-to-sync replication cycle resulting in sync replication data transfer; and
upon completion of the sync replication data transfer, transitioning the state machine to a send_cycle state.
17. The computer program product of claim 16 , wherein the operations further include:
upon determining an occurrence of an IO failure during the async-to-sync replication cycle, transitioning the state machine back to the tripped state.
18. The computer program product of claim 16 , wherein during the send_cycle state, the operations further comprise:
performing async-to-sync transition cycle transfer concurrently with sync replication;
monitoring progress of the async-to-sync transition cycle transfer and sync replication; and
upon completion of the async-to-sync transition cycle transfer and the sync replication, transitioning the state machine back to the sync state.
19. The computer program product of claim 18 , wherein upon determining an occurrence of an IO failure during the async-to-sync transition cycle transfer or the sync replication, transitioning the state machine back to the tripped state.
20. The computer program product of claim 15 , wherein the error condition comprises an IO error condition in which a set timeout value is 30 seconds and the data subject to updating at the target site has not been persisted prior to expiration of the timeout value; and
wherein a plurality of state machines is provided in the storage system, and each of the state machines corresponds to one of a plurality of synchronous replication sessions.