Methods and systems to reduce latency of input/output (I/O) operations during creation of common snapshots based on batching of synchronous replicated datasets at a primary storage system to a cross-site secondary storage system
Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during a common snapshot procedure for storage objects. A computer-implemented method includes initiating a snapshot multi create operation to selectively form a batch of first and second synchronous replicated datasets that belong to a first group of storage disks at the primary storage site and corresponding second group of storage disks at the secondary storage site, performing a batch snapshot create operation on the primary storage site by executing snapshots of storage objects on the primary storage site of the batch of first and second synchronous replicated datasets in parallel multiple threads to effectively utilize processing resources on the primary storage site, and initiating an independent workflow and state machine for each storage object of the batch of first and second synchronous replicated datasets.
1. A computer-implemented method for reducing a resumption time of processing of input/output (I/O) operations during a common snapshot process performed by one or more processors of a multi-site distributed storage system with a primary storage site having a first storage node and a secondary storage site having a second storage node, the computer-implemented method comprising:
initiating a snapshot multi create operation to selectively form a batch of first and second synchronous replicated datasets each having storage objects that belong to a first group of storage disks at the primary storage site and corresponding second group of storage disks at the secondary storage site;
performing a batch snapshot create operation on the primary storage site by executing snapshots of storage objects on the primary storage site of the batch of first and second synchronous replicated datasets in parallel multiple threads to utilize processing resources on the primary storage site; and
initiating an independent workflow and state machine for each storage object of the batch of first and second synchronous replicated datasets.
2. The computer-implemented method of claim 1 , further comprising:
starting a drain and hold state for the primary storage site to stop processing of I/O operations and start queuing incoming write operations, and to complete pending write operations received before the drain and hold state; and
subsequent to the drain and hold state, the batch snapshot create operation is performed on the primary storage site for each storage object of the batch and sending the batch snapshot create operation to the secondary storage site to be performed on each storage object of the batch of the secondary storage site in parallel multiple threads to utilize processing resources on the secondary storage site, wherein the first replicated dataset of the batch has a synchronous replication relationship between a first storage object of the primary storage site and a replicated second storage object of the secondary storage site.
3. The computer-implemented method of claim 1 , further comprising:
tracking pending operations of the primary storage site.
4. The computer-implemented method of claim 2 , wherein the batch snapshot create operation causes creation of a snapshot of the first storage object and a snapshot of the replicated second storage object of the first synchronous replicated dataset.
5. The computer-implemented method of claim 1 , wherein the batch snapshot create operation causes creation of a snapshot of a third storage object and a snapshot of a replicated fourth storage object of the second synchronous replicated dataset.
6. The computer-implemented method of claim 1 , wherein the snapshot multi create operation indicates a batch identifier and a number of storage objects in one or more storage nodes of the batch.
7. The computer-implemented method of claim 1 , further comprising: upon successful completion of the batch snapshot create operation, queued ops are allowed to continue splitting for the first storage node and the second storage node.
8. A multi-site distributed storage system having a primary storage site with a first cluster and a secondary storage site with a second cluster comprising:
one or more processing resources; and
a non-transitory computer-readable medium coupled to the processing resource, having stored therein instructions, which when executed by the one or more processing resources cause the one or more processing resources to:
initiate a snapshot multi create operation to selectively form a batch of first and second synchronous replicated datasets each having storage objects that belong to a first group of storage disks at the primary storage site and corresponding second group of storage disks at the secondary storage site;
perform a batch snapshot create operation on the primary storage site by executing snapshots of storage objects on the primary storage site of the batch of first and second synchronous replicated datasets in parallel multiple threads to utilize processing resources on the primary storage site; and
initiate an independent workflow and state machine for each storage object of the batch of first and second synchronous replicated datasets.
9. The multi-site distributed storage system of claim 8 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
starting a drain and hold state for the primary storage site to stop processing of I/O operations and start queuing incoming write operations, and to complete pending write operations received before the drain and hold state; and
subsequent to the drain and hold state, the batch snapshot create operation is performed on the primary storage site for each storage object of the batch and sending the batch snapshot create operation to the secondary storage site to be performed on each storage object of the batch of the secondary storage site in parallel multiple threads to utilize processing resources on the secondary storage site, wherein the first replicated dataset of the batch has a synchronous replication relationship between a first storage object of the primary storage site and a replicated second storage object of the secondary storage site.
10. The multi-site distributed storage system of claim 8 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
tracking pending operations of the primary storage site.
11. The multi-site distributed storage system of claim 2 , wherein the batch snapshot create operation causes creation of a snapshot of the first storage object and a snapshot of the replicated second storage object of the first synchronous replicated dataset.
12. The multi-site distributed storage system of claim 8 , wherein the batch snapshot create operation causes creation of a snapshot of a third storage object and a snapshot of a replicated fourth storage object of the second synchronous replicated dataset.
13. The multi-site distributed storage system of claim 8 , wherein the snapshot multi create operation indicates a batch identifier and a number of storage objects in one or more storage nodes of the batch.
14. The multi-site distributed storage system of claim 8 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
upon successful completion of the batch snapshot create operation, continue splitting queued ops.
15. A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a storage node cause the one or more processing resources to:
initiate a snapshot multi create operation to selectively form a batch of first and second synchronous replicated datasets each having storage objects that belong to a first group of storage disks at the storage node of a primary storage site and corresponding second group of storage disks at a secondary storage site;
perform a batch snapshot create operation on the primary storage site by executing snapshots of storage objects on the primary storage site of the batch of first and second synchronous replicated datasets in parallel multiple threads to utilize processing resources on the primary storage site; and
initiate an independent workflow and state machine for each storage object of the batch of first and second synchronous replicated datasets.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
start a drain and hold state for the primary storage site to stop processing of I/O operations and start queuing incoming write operations, and to complete pending write operations received before the drain and hold state; and
subsequent to the drain and hold state, the batch snapshot create operation is performed on the primary storage site for each storage object of the batch and sending the batch snapshot create operation to the secondary storage site to be performed on each storage object of the batch of the secondary storage site in parallel multiple threads to utilize processing resources on the secondary storage site, wherein the first replicated dataset of the batch has a synchronous replication relationship between a first storage object of the primary storage site and a replicated second storage object of the secondary storage site.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
tracking pending operations of the primary storage site.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the batch snapshot create operation causes creation of a snapshot of the first storage object and a snapshot of the replicated second storage object of the first synchronous replicated dataset.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the batch snapshot create operation causes creation of a snapshot of a third storage object and a snapshot of a replicated fourth storage object of the second synchronous replicated dataset.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the snapshot multi create operation indicates a batch identifier and a number of storage objects in one or more storage nodes of the batch.