Fast recovery of synchronous replication to minimize disruption in a cross-site distributed storage system

1 . A computer-implemented method performed by one or more processing resources of a distributed storage system, the computer-implemented method comprising:

establishing synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with at least one storage site having read/write access; and

initiating a peer health check to determine a health condition of the primary storage site and concurrently initiating a create snapshot process to create a snapshot of one or more storage objects of the CG2 of the secondary storage site in response to detection of missing heartbeat communications from the primary storage site, wherein the snapshot of the one or more storage objects of the CG2 is utilized to reduce a time for transitioning from out of sync state to in sync state when the out of sync state occurs between the one or more storage objects of CG1 and the one or more storage objects of CG2.

2 . The computer-implemented method of claim 1 , further comprises:

monitoring heartbeat communications between the primary and secondary storage sites.

3 . The computer-implemented method of claim 1 , wherein the snapshot being created is tagged with one or more tags, which indicate that the snapshot is intended for reverse inflight tracking replay and reconciliation from the secondary storage site to the primary storage site.

4 . The computer-implemented method of claim 1 , further comprises:

adding one or more tags to the snapshot of the one or more storage objects of the CG2 of the secondary storage site;

determining whether an automatic unplanned failover (AUFO) occurs based on a result of the health check process;

performing inflight tracking replay and reconciliation between a first Op log of an active file system of the primary storage site and a second Op log that is based upon the snapshot of the secondary storage site if an automatic unplanned failover (AUFO) occurs based on a result of the health check process;

creating a snapshot on the primary storage site with a same version identifier as that of the snapshot created on the secondary storage site; and

performing a resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based upon the snapshot of the primary storage site and the snapshot of the secondary storage site, which is available for the resynchronization process based on removal of the one or more tags from the snapshot of the secondary storage site.

5 . The computer-implemented method of claim 4 , further comprises:

discarding or deleting the snapshot created for the CG2 in case of a false positive with no AUFO being initiated and the one or more tags of the snapshot of the secondary storage site indicating not usable for resynchronization or if the create snapshot process takes beyond a stipulated time period for the health check.

6 . The computer-implemented method of claim 4 , further comprises:

upon the health check resulting in an AUFO action and the create snapshot process is complete, tagging the snapshot from the snapshot creation for the secondary storage site as usable for reverse resynchronization.

7 . The computer-implemented method of claim 4 , wherein the inflight tracking replay and reconciliation is performed upon the completion of the AUFO and when the primary storage site becomes available.

8 . A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a multi-site distributed storage system cause the one or more processing resources to:

establish synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with at least one storage site having read/write access; and

initiate a peer health check to determine a health condition of the primary storage site and concurrently initiate a create snapshot process to create a snapshot of one or more storage objects of the CG2 of the secondary storage site in response to detection of missing heartbeat communications from the primary storage site, wherein the snapshot of the one or more storage objects of the CG2 is utilized to reduce a time for transitioning from out of sync state to in sync state when the out of sync state occurs between the one or more storage objects of CG1 and the one or more storage objects of CG2.

9 . The non-transitory computer-readable storage medium of claim 8 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:

monitor heartbeat communications between the primary and secondary storage sites.

10 . The non-transitory computer-readable storage medium of claim 8 , wherein the snapshot being created is tagged with one or more tags, which indicates that the snapshot is intended for reverse inflight tracking replay and reconciliation from the secondary storage site to the primary storage site.

11 . The non-transitory computer-readable storage medium of claim 8 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:

add one or more tags to the snapshot of the one or more storage objects of the CG2 of the secondary storage site;

determine whether an automatic unplanned failover (AUFO) occurs based on a result of the health check process;

perform inflight tracking replay and reconciliation between a first Op log of an active file system of the primary storage site and a second Op log that is based upon the snapshot of the secondary storage site if an automatic unplanned failover (AUFO) occurs based on a result of the health check process;

create a snapshot on the primary storage site with a same version identifier as that of the snapshot created on the secondary storage site; and

performing a resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based upon the snapshot of the primary storage site and the snapshot of the secondary storage site, which is available for the resynchronization process based on removal of the one or more tags from the snapshot of the secondary storage site.

12 . The non-transitory computer-readable storage medium of claim 11 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:

discard or delete the snapshot created for the CG2 in case of a false positive with no AUFO being initiated and the one or more tags of the snapshot of the CG2 indicating not usable for resynchronization or if the create snapshot process takes beyond a stipulated time period for the health check.

13 . The non-transitory computer-readable storage medium of claim 11 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:

upon the health check resulting in an AUFO action and the create snapshot process is complete, tag the snapshot from the snapshot creation for the secondary storage site as usable for reverse resynchronization.

14 . The non-transitory computer-readable storage medium of claim 11 , wherein the inflight tracking replay and reconciliation is performed upon the completion of the AUFO and when the primary storage site becomes available.

15 . A multi-site distributed storage system having a primary storage site with a primary storage cluster and a secondary storage site with a secondary storage cluster, comprising:

one or more processing resources; and

a non-transitory computer-readable medium coupled to the one or more processing resources, having stored therein instructions, which when executed by the one or more processing resources cause the one or more processing resources to:

establish synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with at least one storage site having read/write access; and

initiate a peer health check to determine a health condition of the primary storage site and concurrently initiate a create snapshot process to create a snapshot of one or more storage objects of the CG2 of the secondary storage site in response to detection of missing heartbeat communications from the primary storage site, wherein the snapshot of the one or more storage objects of the CG2 is utilized to reduce a time for transitioning from out of sync state to in sync state when the out of sync state occurs between the one or more storage objects of CG1 and the one or more storage objects of CG2.

16 . The multi-site distributed storage system of claim 15 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:

monitor heartbeat communications between the primary and secondary storage sites.

17 . The multi-site distributed storage system of claim 15 , wherein the snapshot being created is tagged with one or more tags, which indicates that the snapshot is intended for reverse inflight tracking replay and reconciliation from the secondary storage site to the primary storage site.

18 . The multi-site distributed storage system of claim 15 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:

add one or more tags to the snapshot of the one or more storage objects of the CG2 of the secondary storage site;

determine whether an automatic unplanned failover (AUFO) occurs based on a result of the health check process;

perform inflight tracking replay and reconciliation between a first Op log of an active file system of the primary storage site and a second Op log that is based upon the snapshot of the secondary storage site if an automatic unplanned failover (AUFO) occurs based on a result of the health check process;

create a snapshot on the primary storage site with a same version identifier as that of the snapshot created on the secondary storage site; and

performing a resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based upon the snapshot of the primary storage site and the snapshot of the secondary storage site, which is available for the resynchronization process based on removal of the one or more tags from the snapshot of the secondary storage site.

19 . The multi-site distributed storage system of claim 18 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:

discard or delete the snapshot created for the CG2 in case of a false positive with no AUFO being initiated and the one or more tags of the snapshot indicating not usable for resynchronization or if the snapshot creation takes beyond a stipulated time period for the health check.

20 . The multi-site distributed storage system of claim 18 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:

upon the health check resulting in an AUFO action and the snapshot creation is complete, tag the snapshot from the create snapshot process for the secondary storage site as usable for reverse resynchronization.