Sand timer algorithm for tracking in-flight data storage requests for data replication
A method is disclosed comprising: generating a first snapshot of a first storage subsystem; detecting, by a management node, that all in-flight data storage requests recorded in drain tables of storage nodes in the first storage subsystem have been completed, the in-flight data storage requests recorded in the drain tables of the storage nodes being replicated in a second storage subsystem; causing, by the management node, each of the storage nodes to flip the respective designations of the tracking tables in the node's respective pair of tracking tables; and transmitting, from the management node to the second storage subsystem, an instruction which when received by the second storage subsystem causes the second storage subsystem to generate a second snapshot of the second storage subsystem.
1. A method for use in a storage node, the method comprising:
storing, in a memory of the storage node, a pair of tracking tables, each of the tracking tables being arranged to alternate between an active table designation and a drain table designation, each of the tracking tables in the pair being arranged to have a different designation than the other tracking table in the pair;
recording all incoming data storage requests that are received at the storage node in one of the tracking tables that is currently designated as an active tracking table of the storage node;
flipping the respective designations of the tracking tables;
removing any remaining data storage requests, from one of the tracking tables that is currently designated as a drain table of the storage node, when the remaining data storage requests are completed;
transmitting, to a management node, a notification that the drain table of the storage node is empty, the notification being transmitted after all of the remaining data storage requests have been removed from one of the tracking tables that is currently designated as the drain table of the storage node.
2. The method of claim 1 , wherein the respective designations of the tracking tables are flipped in response to an instruction from the management node.
3. The method of claim 1 , wherein recording all incoming data storage requests in one of the tracking tables that is currently designated as the active tracking table of the storage node includes adding respective identifiers corresponding to the incoming data storage requests to the tracking table that is currently designated as the active tracking table.
4. The method of claim 1 , wherein incoming data storage requests are recorded in any given one of the tracking tables in the pair only during periods when the given tracking table is designated as an active table of the storage node.
5. The method of claim 1 , wherein flipping the respective designations of the tracking tables includes: (i) changing the respective designation of one of the tracking tables in the pair from the drain table designation to the active table designation, and (ii) changing the respective designation of the other one of the tracking tables in the pair from the active table designation to the drain table designation.
6. The method of claim 1 , further comprising detecting whether one of the tracking tables that is currently designated as the drain table is empty.
7. The method of claim 1 , wherein the storage node includes a control node of a storage system, and each of the incoming data storage requests includes a control node command that is generated by any of a plurality of routing nodes in the storage system.
8. A system comprising:
a memory; and
at least one processor that is operatively coupled to the memory, the at least one processor being configured to perform the operations of:
storing, in the memory, a pair of tracking tables, each of the tracking tables being arranged to alternate between a tracking table designation and a drain table designation, each of the tracking tables in the pair being arranged to have a different designation than the other tracking table in the pair;
recording any incoming data storage requests in one of the tracking tables that is currently designated as an active tracking table;
flipping the respective designations of the tracking tables;
removing any remaining data storage requests, from one of the tracking tables that is currently designated as a drain table, when the remaining data storage requests are completed;
transmitting, to a management node, a notification that one of the tracking tables that is currently designated as the drain table is empty, the notification being transmitted after all of the remaining data storage requests have been removed from one of the tracking tables that is currently designated as the drain table.
9. The system of claim 8 , wherein the respective designations of the tracking tables are flipped in response to an instruction from the management node.
10. The system of claim 8 , wherein recording any incoming data storage requests in one of the tracking tables that is currently designated as the active tracking table includes adding respective identifiers corresponding to the incoming data storage requests to the tracking table that is currently designated as the active tracking table.
11. The system of claim 8 , wherein incoming data storage requests are recorded in any given one of the tracking tables in the pair only during periods when the given tracking table is designated as an active table.
12. The system of claim 8 , wherein flipping the respective designations of the tracking tables includes: (i) changing the respective designation of one of the tracking tables in the pair from the drain table designation to the active table designation, and (ii) changing the respective designation of the other one of the tracking tables in the pair from the active table designation to the drain table designation.
13. The system of claim 8 , wherein the at least one processor is further configured to perform the operation of detecting whether one of the tracking tables that is currently designated as the drain table is empty.
14. The system of claim 8 , wherein the processor is configured to execute a control node of a storage system, and each of the incoming data storage requests includes a control node command that is generated by any of a plurality of routing nodes in the storage system.
15. A non-transitory computer-readable medium configured to store one or more processor-executable instructions which when executed by at least one processor cause the at least one processor to perform the operations of:
storing, in a memory of a storage node, a pair of tracking tables, each of the tracking tables being arranged to alternate between a tracking table designation and a drain table designation, each of the tracking tables in the pair being arranged to have a different designation than the other tracking table in the pair;
recording all incoming data storage requests that are received at the storage node in one of the tracking tables that is currently designated as an active tracking table of the storage node;
flipping the respective designations of the tracking tables;
removing any remaining data storage requests, from one of the tracking tables that is currently designated as a drain table of the storage node, when the remaining data storage requests are completed;
transmitting, to a management node, a notification that the drain table of the storage node is empty, the notification being transmitted after all of the remaining data storage requests have been removed from one of the tracking tables that is currently designated as the drain table of the storage node.
16. The non-transitory computer-readable medium of claim 15 , wherein the respective designations of the tracking tables are flipped in response to an instruction from the management node.
17. The non-transitory computer-readable medium of claim 15 , wherein recording all incoming data storage requests in one of the tracking tables that is currently designated as the active tracking table of the storage node includes adding respective identifiers corresponding to the incoming data storage requests to the tracking table that is currently designated as the active tracking table.
18. The non-transitory computer-readable medium of claim 15 , wherein incoming data storage requests are recorded in any given one of the tracking tables in the pair only during periods when the given tracking table is designated as an active table.
19. The non-transitory computer-readable medium of claim 15 , wherein flipping the respective designations of the tracking tables includes: (i) changing the respective designation of one of the tracking tables in the pair from the drain table designation to the active table designation, and (ii) changing the respective designation of the other one of the tracking tables in the pair from the active table designation to the drain table designation.
20. The non-transitory computer-readable medium of claim 15 , wherein the storage node includes a control node of a storage system, and each of the incoming data storage requests includes a control node command that is generated by any of a plurality of routing nodes in the storage system.