Consistent backups based on local node clock
Disclosed are systems and methods for generating consistent backups. A central coordinator informs each node storing a partition of the time to perform a backup. At the designated time, each node blocks updates for a corresponding time interval measured by its local clock. Each node performs the backup operation according to its own local clock. Consistent backups may be generated in spite of clock skew between the local clocks as long as the time interval is at least as long as a maximum local clock skew (among the nodes performing the backup). In some systems the maximum local clock skew may be reduced, by a round-trip update latency from a client, for example.
1. A system, comprising:
a plurality of storage nodes, each of which comprises one or more storage devices or logical storage volumes of a data storage service that stores data in partitions; and
a central coordinator to send an indication of a time to perform a backup operation for a data set to each of the nodes;
each of the storage nodes to:
receive the indication of the time to perform the backup operation for the data set;
block updates to the respective partition of the data set at the respective node for a time interval based on the indicated time as measured by a local clock of the respective node, wherein the time interval;
is at least partially concurrent across the plurality of nodes, and
is shortened by an amount of time equivalent to or based on a minimum round-trip latency for the plurality of nodes to send to a client an acknowledgement of an update to the data set and receive a subsequent update to the data set sent from the client after the client received the acknowledgment; and
perform the backup operation during the time interval at the respective node for its respective partition of the data set.
2. The system as recited in claim 1 , wherein the time interval is based at least in part on an indication of maximum clock skew between the local clocks of the plurality of nodes.
3. The system as recited in claim 1 , wherein said perform the backup operation during the time interval at the respective node for its respective partition of the data set is triggered without a trigger from the central coordinator.
4. A method, comprising:
receiving, at each node of a plurality of nodes storing respective partitions of a data set, an indication of a time to perform a backup operation for the data set;
blocking updates to the respective partitions of the data set at each node of the plurality of nodes for a time interval based on the indicated time as measured by a local clock of the respective node, wherein the time interval;
is at least partially concurrent across the plurality of nodes, and
is shortened by an indication of a minimum round-trip latency for the plurality of nodes to send to a client an acknowledgement of an update to the data set and receive a subsequent update to the data set sent from the client after the client received the acknowledgment; and
performing the backup operation for the indicated time at each node for the respective partitions of the data set.
5. The method as recited in claim 4 , wherein the time interval is based at least in part on an indication of maximum clock skew between the local clocks of the plurality of nodes.
6. The method as recited in claim 4 , wherein said performing the backup operation for the indicated time at each node for the respective partitions of the data set comprises performing the backup during the time interval as measured by the local clocks of respective nodes.
7. The method as recited in claim 4 , wherein said blocking updates comprises:
buffering the updates until the time interval expires as measured by the local clock of the respective node; and
performing the updates held in buffer subsequent to expiration of the time interval as measured by the local clock of the respective node.
8. The method as recited in claim 4 , wherein said blocking updates comprises:
performing the update; and
either not sending a corresponding acknowledgement for the update, or pausing for a period of time before sending the corresponding acknowledgement for the update.
9. The method recited in claim 4 , further comprising
receiving the updates from one or more clients; and
performing each of said blocking and said performing the backup operation without the one or more clients propagating consistency information with the update requests.
10. The method recited in claim 4 , further comprising:
identifying that clock skew of a local clock of a host for one of the plurality of nodes is above a clock-skew threshold;
moving a partition corresponding to the one node to a host with a clock skew less than the clock-skew threshold.
11. A non-transitory computer readable storage medium storing program instruction that, if executed by a computer, cause the computer to perform:
receiving, at each node of a plurality of nodes storing respective partitions of a data set, an indication of a time to perform a backup operation for the data set;
blocking updates to the respective partitions of the data set at each node of the plurality of nodes for a time interval based on the indicated time as measured by a local clock of the respective node, wherein the time interval;
is at least partially concurrent across the plurality of nodes, and
is shortened by an indication of a minimum round-trip latency for the plurality of nodes to send to a client an acknowledgement of an update to the data set and receive a subsequent update to the data set sent from the client after the client received the acknowledgment; and
performing the backup operation for the indicated time at each node for the respective partitions of the data set.
12. The non-transitory computer readable storage medium as in claim 11 , wherein the time interval is based at least in part on an indication of maximum clock skew between the local clocks of the plurality of nodes.
13. The non-transitory computer readable storage medium as in claim 11 , wherein to perform said performing the backup operation for the indicated time at each node for the respective partitions of the data set, the program instructions cause the computer to perform performing the backup during the time interval as measured by the local clocks of respective nodes.
14. The non-transitory computer readable storage medium as in claim 11 , where to perform said blocking updates, the program instructions, when executed by the computer, further cause the computer to perform:
buffering the updates until the time interval expires as measured by the local clock of the respective node; and
performing the updates held in buffer subsequent to expiration of the time interval as measured by the local clock of the respective node.
15. The non-transitory computer readable storage medium as in claim 11 , wherein to perform said blocking updates, the program instructions, when executed by the computer, further cause the computer to perform:
performing the update; and
either not sending a corresponding acknowledgement for the update, or pausing for a period of time before sending the corresponding acknowledgement for the update.
16. The non-transitory computer readable storage medium as in claim 11 , wherein the program instructions further cause the computer to perform:
receiving the updates from one or more clients; and
performing each of said blocking and said performing the backup operation without the one or more clients propagating consistency information with the update requests.
17. The non-transitory computer readable storage medium as in claim 11 , wherein the program instructions further cause the computer to perform:
identifying that clock skew of a local clock of a host for one of the plurality of nodes is above a clock-skew threshold; and
moving a partition corresponding to the one node to a host with a clock skew less than the clock-skew threshold.