Performing various operations at the granularity of a consistency group within a cross-site storage solution
Systems and methods are described for efficiently performing various operations at the granularity of a consistency group (CG) within a cross-site storage solution. An example of one of the various operations includes an independent and parallel resynchronization approach that independently brings individual volumes of a CG to a steady state of in-synchronization (InSync), thereby contributing to scalability of CGs by supporting CGs having a large number of member volumes without requiring a change to the resynchronization process. Another example includes preserving dependent write-order consistency when a remote mirror copy goes out-of-synchronization (OOS) for any reason by driving all member volumes OOS responsive to any member volume becoming OOS. Yet another example includes independent creation of snapshots by member volumes to support efficient and on-demand creation by an application of a common snapshots of all or a subset of peered member volumes of a CG with which the application is associated.
1. A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a distributed storage system, cause the distributed storage system to:
detect a first volume of a plurality of volumes that are members of a local consistency group of the distributed storage system is in an out-of-synchronization (OOS) state with respect to a mirror copy of a dataset associated with the local consistency group that is maintained on a plurality of corresponding volumes that are members of a remote consistency group of a remote distributed storage system; and
preserve dependent write-order consistency on the mirror copy by:
temporarily disallowing access to data stored on the first volume;
broadcasting, on behalf of the first volume, a first request of a three-way handshake of a consensus protocol to the local consistency group, wherein the consensus protocol is for solving a problem of consensus in a distributed system;
responsive to receipt of the first request directed to a given volume of the plurality of volumes:
temporarily disallowing access to data stored on the given volume; and
triggering a second request of the three-way handshake of the consensus protocol by acknowledging the first request; and
responsive to receipt of the second request directed to a particular volume of the plurality of volumes, placing the particular volume in the OOS state and permitting access to the data stored on the particular volume.
2. The non-transitory computer-readable storage medium of claim 1 , wherein said temporarily disallowing access to data stored on the first volume comprises holding responses of completed input/output operations until a distributed transaction associated with the three-way handshake of the consensus protocol has been completed.
3. The non-transitory computer-readable storage medium of claim 1 , wherein said temporarily disallowing access to data stored on the first volume comprises holding input/output operations until a distributed transaction associated with the three-way handshake of the consensus protocol has been completed.
4. The non-transitory computer-readable storage medium of claim 1 , wherein the first request of the three-way handshake comprises a phase 1 request.
5. The non-transitory computer-readable storage medium of claim 1 , wherein the second request of the three-way handshake comprises a phase 2 request.
6. The non-transitory computer-readable storage medium of claim 1 , wherein the three-way handshake of the consensus protocol causes all other members of the local consistency group to transition into the OOS state.
7. A distributed storage system comprising:
one or more processing resources; and
one or more non-transitory computer-readable media, coupled to the one or more processing resources, having stored therein instructions that when executed by the one or more processing resources cause the distributed storage system to:
detect a first volume of a plurality of volumes that are members of a local consistency group of the distributed storage system is in an out-of-synchronization (OOS) state with respect to a mirror copy of a dataset associated with the local consistency group that is maintained on a plurality of corresponding volumes that are members of a remote consistency group of a remote distributed storage system; and
preserve dependent write-order consistency on the mirror copy by:
temporarily disallowing access to data stored on the first volume;
broadcasting, on behalf of the first volume, a first request of a three-way handshake of a consensus protocol to the local consistency group, wherein the consensus protocol is for solving a problem of consensus in a distributed system;
responsive to receipt of the first request directed to a given volume of the plurality of volumes:
temporarily disallowing access to data stored on the given volume; and
triggering a second request of the three-way handshake of the consensus protocol by acknowledging the first request; and
responsive to receipt of the second request directed to a particular volume of the plurality of volumes, placing the particular volume in the OOS state and permitting access to the data stored on the particular volume.
8. The distributed storage system of claim 7 , wherein said temporarily disallowing access to data stored on the first volume comprises holding responses of completed input/output operations until a distributed transaction associated with the three-way handshake of the consensus protocol has been completed.
9. The distributed storage system of claim 7 , wherein said temporarily disallowing access to data stored on the first volume comprises holding input/output operations until a distributed transaction associated with the three-way handshake of the consensus protocol has been completed.
10. The distributed storage system of claim 7 , wherein the first request of the three-way handshake comprises a phase 1 request.
11. The distributed storage system of claim 7 , wherein the second request of the three-way handshake comprises a phase 2 request.
12. The distributed storage system of claim 7 , wherein the three-way handshake of the consensus protocol causes all other members of the local consistency group to transition into the OOS state.
13. A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a distributed storage system, cause the distributed storage system to:
receive a request to create a common snapshot between a local consistency group including a first plurality of volumes of the distributed storage system and a remote consistency group including a second plurality of volumes of a remote distributed storage system, wherein the remote consistency group is protecting the local consistency group;
initiating, on behalf of a first volume of the first plurality of volumes, a first three-way handshake of a consensus protocol with the local consistency group;
responsive to receipt, on behalf of a given volume of the first plurality of volumes, of a final request of the first three-way handshake:
temporarily disallow access to data stored on the given volume;
cause a first snapshot of the data stored on the given volume to be created in a dependent write-order consistent manner independent of any other volumes of the first plurality of volumes and in parallel with one or more other volumes of the first plurality of volumes;
cause a second snapshot of data stored on a volume of the second plurality of volumes corresponding to the given volume to be created in a dependent write-order consistent manner independent of any other volumes of the second plurality of volumes and in parallel with one or more other volumes of the second plurality of volumes; and
after the first snapshot and the second snapshot have been created, send a response to the first volume;
responsive to receipt, on behalf of the first volume, of responses from or on behalf of all other volumes of the first plurality of volumes, cause each of the first plurality of volumes to start allowing access to their respective data by initiating a second three-way handshake of the consensus protocol with the local consistency group.
14. The non-transitory computer-readable storage medium of claim 13 , wherein execution of the instructions by the one or more processing resources further cause the distributed storage system to responsive to receipt of the final request of the first three-way handshake, synchronize in-flight data between a subset of volumes of the first plurality of volumes that are associated with a common set of one or more disks and a corresponding subset of volumes of the second plurality of volumes by flushing a file system buffer cache for each volume of the subset of volumes to the common set of disks.
15. The non-transitory computer-readable storage medium of claim 13 , wherein the first plurality of volumes comprises a subset of all member volumes of the local consistency group.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the distributed storage system comprises a plurality of storage nodes organized as a cluster.
17. A distributed storage system comprising:
one or more processing resources; and
one or more non-transitory computer-readable media, coupled to the one or more processing resources, having stored therein instructions that when executed by the one or more processing resources cause the distributed storage system to:
receive a request to create a common snapshot between a local consistency group including a first plurality of volumes of the distributed storage system and a remote consistency group including a second plurality of volumes of a remote distributed storage system, wherein the remote consistency group is protecting the local consistency group;
initiating, on behalf of a first volume of the first plurality of volumes, a first three-way handshake of a consensus protocol with the local consistency group;
responsive to receipt, on behalf of a given volume of the first plurality of volumes, of a final request of the first three-way handshake:
temporarily disallow access to data stored on the given volume;
cause a first snapshot of the data stored on the given volume to be created in a dependent write-order consistent manner independent of any other volumes of the first plurality of volumes and in parallel with one or more other volumes of the first plurality of volumes;
cause a second snapshot of data stored on a volume of the second plurality of volumes corresponding to the given volume to be created in a dependent write-order consistent manner independent of any other volumes of the second plurality of volumes and in parallel with one or more other volumes of the second plurality of volumes; and
after the first snapshot and the second snapshot have been created, send a response to the first volume;
responsive to receipt, on behalf of the first volume, of responses from or on behalf of all other volumes of the first plurality of volumes, cause each of the first plurality of volumes to start allowing access to their respective data by initiating a second three-way handshake of the consensus protocol with the local consistency group.
18. The distributed storage system of claim 17 , wherein execution of the instructions by the one or more processing resources further cause the distributed storage system to responsive to receipt of the final request of the first three-way handshake, synchronize in-flight data between a subset of volumes of the first plurality of volumes that are associated with a common set of one or more disks and a corresponding subset of volumes of the second plurality of volumes by flushing a file system buffer cache for each volume of the subset of volumes to the common set of disks.
19. The distributed storage system of claim 17 , wherein the first plurality of volumes comprises a subset of all member volumes of the local consistency group.
20. The distributed storage system of claim 17 , wherein the distributed storage system further comprises a plurality of storage nodes organized as a cluster.