Sand timer algorithm for tracking in-flight data storage requests for data replication

1. A method for use in a first storage subsystem having a management node and a plurality of storage nodes, each of the storage nodes having a respective pair of tracking tables for recording in-flight data storage requests, the tracking tables in each pair having different respective designations, the tracking tables in each pair alternating between a drain table designation and an active table designation, the method comprising:

generating a first snapshot of the first storage subsystem;

detecting, by the management node, that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed, the in-flight data storage requests recorded in the drain tables of the storage nodes being replicated in a second storage subsystem;

causing, by the management node, each of the plurality of storage nodes to flip the respective designations of the tracking tables in the node's respective pair of tracking tables; and

transmitting, from the management node to the second storage subsystem, an instruction which when received by the second storage subsystem causes the second storage subsystem to generate a second snapshot of the second storage subsystem, the instruction being transmitted after the first snapshot is generated and all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed.

2. The method of claim 1 , further comprising:

detecting that a replication session of the first storage subsystem and the second storage subsystem is out of sync; and

transmitting to the second storage subsystem a difference between at least one production object of the first storage subsystem and the first snapshot of the first storage subsystem.

3. The method of claim 1 , wherein the in-flight data storage requests recorded in the drain tables of the storage nodes are synchronously replicated to the second storage subsystem.

4. The method of claim 1 , wherein detecting that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed includes receiving a different respective message from each one the storage nodes indicating that all data storage requests recorded in the storage node's respective drain table have been completed.

5. The method of claim 1 , wherein causing each of the storage nodes to flip the respective designations of the tracking tables in the node's respective pair of tracking tables includes transmitting to each of the plurality of storage nodes a respective instruction which when received by the storage node causes the storage node to: (i) designate the storage node's current active table as drain table, and (ii) designate the storage node's current drain table as active table.

6. The method of claim 1 , wherein recording an in-flight data storage request in any of the tracking tables includes adding an identifier corresponding to the in-flight data storage request to the tracking table, such that identifiers corresponding to in-flight data storage requests are added to any given one of the tracking tables only during periods when the given tracking table carries active table designation.

7. An apparatus for use in a first storage subsystem having a plurality of storage nodes, each of the storage nodes having a respective pair of tracking tables for recording in-flight data storage requests, the tracking tables in each pair having different respective designations, the tracking tables in each pair alternating between a drain table designation and an active table designation, the apparatus comprising:

a memory; and

at least one processor operatively coupled to the memory, the at least one processor being configured to:

generate a first snapshot of the first storage subsystem;

detect that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed, the in-flight data storage requests recorded in the drain tables of the storage nodes being replicated in a second storage subsystem;

cause each of the plurality of storage nodes to flip the respective designations of the tracking tables in the node's respective pair of tracking tables; and

transmit, to the second storage subsystem, an instruction which when received by the second storage subsystem causes the second storage subsystem to generate a second snapshot of the second storage subsystem, the instruction being transmitted after the first snapshot is generated and all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed.

8. The apparatus of claim 7 , wherein the at least one processor is further configured to:

detect that a replication session of the first storage subsystem and the second storage subsystem is out of sync; and

transmit to the second storage subsystem a difference between at least one production object of the first storage subsystem and the first snapshot of the first storage subsystem.

9. The apparatus of claim 7 , wherein the in-flight data storage requests recorded in the drain tables of the storage nodes are synchronously replicated to the second storage subsystem.

10. The apparatus of claim 7 , wherein detecting that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed includes receiving a different respective message from each one the storage nodes indicating that all data storage requests recorded in the storage node's respective drain table have been completed.

11. The apparatus of claim 7 , wherein causing each of the storage nodes to flip the respective designations of the tracking tables in the node's respective pair of tracking tables includes transmitting to each of the plurality of storage nodes a respective instruction which when received by the storage node causes the storage node to: (i) designate the storage node's current active table as drain table, and (ii) designate the storage node's current drain table as active table.

12. The apparatus of claim 7 , wherein recording an in-flight data storage request in any of the tracking tables includes adding an identifier corresponding to the in-flight data storage request to the tracking table, such that identifiers corresponding to in-flight data storage requests are added to any given one of the tracking tables only during periods when the given tracking carries the active table designation.

13. A non-transitory computer-readable medium configured to store one or more processor-executable instructions which when executed by at least one processor cause the at least one processor to perform the operations of:

identifying a plurality of storage nodes in a first storage subsystem, each of the storage nodes having a respective pair of tracking tables for recording in-flight data storage requests, the tracking tables in each pair having different respective designations, the tracking tables in each pair alternating between a drain table designation and an active table designation;

generating a first snapshot of the first storage subsystem;

detecting that all in-flight data storage requests recorded in drain tables of the storage nodes have been completed, the in-flight data storage requests recorded in the drain tables of the storage nodes being replicated in a second storage subsystem;

causing each of the plurality of storage nodes to flip the respective designations of the tracking tables in the node's respective pair of tracking tables; and

transmitting to the second storage subsystem, an instruction which when received by the second storage subsystem causes the second storage subsystem to generate a second snapshot of the second storage subsystem, the instruction being transmitted after the first snapshot is generated and all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed.

14. The non-transitory computer-readable medium of claim 13 , wherein the processor-executable instructions, when executed by the at least one processor, further cause the at least one processor to perform the operations of:

detecting that a replication session of the first storage subsystem and the second storage subsystem is out of sync; and

transmitting to the second storage subsystem a difference between at least one production object of the first storage subsystem and the first snapshot of the first storage subsystem.

15. The non-transitory computer-readable medium of claim 13 , wherein the in-flight data storage requests recorded in the drain tables of the storage nodes are synchronously replicated to the second storage subsystem.

16. The non-transitory computer-readable medium of claim 13 , wherein detecting that all in-flight data storage requests recorded in the drain tables of the storage nodes have been completed includes receiving a different respective message from each one the storage nodes indicating that all data storage requests recorded in the storage node's respective drain table have been completed.

17. The non-transitory computer-readable medium of claim 13 , wherein causing each of the storage nodes to flip the respective designations of the tracking tables in the node's respective pair of tracking tables includes transmitting to each of the plurality of storage nodes a respective instruction which when received by the storage node causes the storage node to: (i) designate the storage node's current active table as drain table, and (ii) designate the storage node's current drain table as active table.

18. The non-transitory computer-readable medium of claim 13 , wherein recording an in-flight data storage request in any of the tracking tables includes adding an identifier corresponding to the in-flight data storage request to the tracking table, such that identifiers corresponding to in-flight data storage requests are added to any given one of the tracking tables only during periods when the given tracking table carries the active table designation.