System and method for replication using consistency interval markers in a distributed storage environment
View Patent ↗Applications executing on various nodes in a distributed storage environment may write data to primary storage and may also replicate the data to secondary storage via a replication target. An interval coordinator may coordinate the periodic saving of checkpoints or snapshots of the replicated data. The interval coordinator may determine the length of consistency intervals between the saving of each of the checkpoints. Writes to the replication target from each of the nodes may be associated with the current consistency interval and, in some embodiments, with a unique per-node sequence number. When transitioning between consistency intervals, each node may be configured to temporarily suspend completion of the writes and to send the replication target a consistency interval marker indicating that the node has completed all writes for the current consistency interval.
1. A method, comprising:
one or more nodes in a distributed computing environment sending replicated write requests to a replication target device, wherein an indication of a first consistency interval is associated with each replicated write request;
initiating a consistency interval transition that defines a point in time for which data on the replication target device is in a consistent state;
in response to the initiation of the consistency interval transition, each of the one or more nodes:
temporarily suspending sending replicated write requests to the replication target device; and subsequent to said temporarily suspending:
sending a consistency interval marker to the replication target device; and
resuming sending replicated write requests to the replication target device, wherein an indication of a second consistency interval is associated with each replicated write request sent after said resuming.
2. The method of claim 1 , wherein each of the replicated write requests further includes a node-specific sequence number, wherein the node-specific sequence number is incremented with each replicated write request from a single one of the nodes.
3. The method of claim 1 , wherein initiating an interval transition comprises a consistency interval coordinator sending an interval end message to each of the nodes including an indication that the first consistency interval has ended.
4. The method of claim 3 , further comprising each of the nodes sending an interval completion acknowledgement message to the consistency interval coordinator in response to receiving the interval end message.
5. The method of claim 4 , further comprising the consistency interval coordinator resolving an ordering ambiguity between two or more write requests from different nodes during the first consistency interval.
6. The method of claim 4 , further comprising the consistency interval coordinator sending an interval start message to each of the nodes in response to receiving an interval completion acknowledgement message from every one of the one or more the nodes.
7. The method of claim 6 , wherein the interval start message includes the indication of the second consistency interval.
8. The method of claim 3 , wherein the consistency interval coordinator resides on the replication target device.
9. The method of claim 3 , further comprising the replication target device sending the consistency interval coordinator a message indicating that the replication target device has received the consistency interval marker from every one of the one or more nodes.
10. The method of claim 1 , further comprising the replication target device storing a checkpoint of data associated with the replicated write requests received from the one or more nodes in response to receiving the consistency interval marker from every one of the one or more nodes.
11. The method of claim 1 , further comprising maintaining consistency recovery information for data associated by the replicated write requests.
12. A system, comprising:
one or more source nodes in a networked computing environment;
a consistency interval coordinator coupled to the one or more source nodes;
a replication target device coupled to the one or more source nodes and the consistency interval coordinator;
wherein each of the one or more source nodes is configured to send replicated write requests to the replication target device, wherein an indication of a first consistency interval is associated with each of the replicated write requests;
wherein the consistency interval coordinator is configured to send an interval end message to each of the one or more source nodes indicating that the first consistency interval has ended;
wherein in response to receiving the interval end message, each of the one or more source nodes is further configured to:
temporarily suspend sending replicated write requests to the replication target device;
send a consistency interval marker to the replication target device; and
resume sending replicated write requests to the replication target device, wherein an indication of a second consistency interval is associated with each replicated write request sent after said resuming;
wherein each of the one or more source nodes, the consistency interval coordinator, and the replication target device are implemented on a hardware device.
13. The system of claim 12 , wherein each of the replicated write requests further includes a node-specific sequence number, wherein the node-specific sequence number is incremented with each replicated write request from a single one of the source nodes.
14. The system of claim 12 , wherein each of the source nodes is further configured to send an interval completion acknowledgement message to the consistency interval coordinator in response to receiving the interval end message.
15. The system of claim 14 , wherein the consistency interval coordinator is further configured to resolve an ordering ambiguity between two or more write requests from different nodes during the first consistency interval.
16. The system of claim 14 , wherein the consistency interval coordinator is further configured to send an interval start message to each of the nodes in response to receiving an interval completion acknowledgement message from every one of the one or more the source nodes.
17. The system of claim 16 , wherein the interval start message includes the indication of the second consistency interval.
18. The system of claim 12 , wherein the consistency interval coordinator resides on the replication target device.
19. The system of claim 12 , further comprising the replication target device sending the consistency interval coordinator a message indicating that the replication target device has received the consistency interval marker from every one of the one or more source nodes.
20. The system of claim 12 , wherein the replication target device is further configured to store a checkpoint of data associated with the replicated write requests received from the one or more source nodes in response to receiving the consistency interval marker from every one of the one or more nodes.
21. A device, comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory comprises program instructions executable by the one or more processors to implement a consistency interval coordinator configured to:
send a consistency interval end message to one or more source nodes in a networked computing environment, wherein each of the one or more source nodes is configured to send replicated write requests to a replication target device, wherein an indication of a first consistency interval is associated with each replicated write request;
receive a message from each of the one or more source nodes acknowledging receipt of the consistency interval end message; and
in response to receiving the message from every one of the one or more source nodes, sending an interval start message comprising an indication of a second consistency interval.
22. A device, comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory comprises program instructions executable by the one or more processors to implement a source node in a network computing environment, wherein the source node is configured to:
send replicated write requests to a replication target device, wherein each replicated write request includes an indication of a first consistency interval;
receive an interval end message from a consistency interval coordinator, wherein the interval end message indicates that the first consistency interval has ended;
in response to receiving the interval end message, the source node is further configured to:
temporarily suspend sending replicated write requests to the replication target device;
send a consistency interval transition message to the replication target device; and
resume sending replicated write requests to the replication target device, wherein each replicated write request sent after said resuming includes an indication of a second consistency interval.
23. A device, comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory comprises program instructions executable by the one or more processors to implement a replication target device in a networked computing environment, wherein the replication target device is configured to:
receive replicated write requests from one or more source nodes in the networked computing environment, wherein an indication of a first consistency interval is associated with each replicated write request;
receive a consistency interval marker from each of the one or more source nodes; and
in response to receiving the consistency interval marker from every one of the source nodes, store a checkpoint of data associated with the replicated write requests received from the one or more source nodes.
24. A computer accessible medium, comprising program instructions computer-executable to implement:
a node in a distributed computing environment sending replicated write requests to a replication target device, wherein an indication of a first consistency interval is associated with each replicated write request;
in response to an initiation of a consistency interval transition, the node:
temporarily suspending sending replicated write requests to the replication target device; and
subsequent to said temporarily suspending:
sending a consistency interval marker to the replication target device; and
resuming sending replicated write requests to the replication target device, wherein an indication of a second consistency interval is associated with each replicated write request sent after said resuming.