JOURNALING FOR SCALEOUT SYSTEMS
According to various embodiments, techniques and mechanisms described herein may facilitate the resynchronization of storage container nodes within a storage volume. In some implementations, a virtual storage volume may be created by aggregating storage resources from two or more storage container nodes. Each storage container node may include a privileged storage container that runs atop a virtualization layer. For redundancy, a virtual storage volume may store the same data on two or more of the storage nodes that make up the volume. However, the data may become out-of-sync, for instance if one or more of the nodes fails during the execution of a storage operation. Data may be resynchronized after a node failure by designating data as source data for resynchronization based on comparing metadata across nodes in view of data integrity guarantees.
1 . A method comprising:
receiving a request to resynchronize a designated data portion stored on two or more storage container nodes, the two or more storage container nodes each configured to store the designated data portion;
comparing a plurality of checksum values for equality, each checksum value being associated with a respective one of the storage container nodes, with a respective timestamp, and with a respective version of the designated data portion;
when it is determined that at least a first one of the plurality of checksum values does not match at least a second one of the plurality of checksum values, identifying a designated one of the checksum values based on the timestamps; and
updating the two or more storage container nodes to each include the designated data portion.
2 . The method recited in claim 1 , wherein identifying the designated checksum value comprises identifying the latest timestamp associated with a checksum value present on any storage container node.
3 . The method recited in claim 1 , the method further comprising:
determining whether the system is configured to enforce stable data writes.
4 . The method recited in claim 3 , wherein identifying the designated checksum value comprises identifying the latest timestamp associated with a checksum value present on a designated number of the storage container nodes when it is determined that the system is configured to enforce stable writes.
5 . The method recited in claim 4 , wherein the designated number of the storage container nodes comprises a majority of the storage container nodes.
6 . The method recited in claim 1 , the method further comprising:
receiving a request for the designated data portion; and
transmitting the designated data portion in response to the request.
7 . The method recited in claim 6 , wherein the storage container nodes are members of a storage container node cluster, the storage container node cluster including a plurality of data storage volumes, the plurality of data storage volumes including a designated data storage volume, the designated data storage volume including the two or more storage container nodes.
8 . The method recited in claim 6 , wherein the request for the designated data portion is generated at a container application, the container application being located at one of the storage container nodes.
9 . The method recited in claim 8 , wherein the request for the designated data portion is generated at a container application, the container application being located at an application node outside the designated data storage volume.
10 . The method recited in claim 1 , wherein each storage container node includes a storage container and a container engine, the container engine being configured to provide a virtualized operating system mediating communications associated with one or more containerized applications.
11 . The method recited in claim 1 , wherein each storage container node includes a respective storage container configured to access a respective one or more storage devices accessible via the storage container node.
12 . A system comprising:
a plurality of computing devices, each computing device including memory and at least one processor, each computing device including a network interface, the computing devices being configured to communicate over a network via the network interfaces, each computing device being configured to implement a storage container node, wherein a designated one of the storage container nodes is configured to:
receive a request to resynchronize a designated data portion stored on two or more of the storage container nodes, the two or more storage container nodes each configured to store the designated data portion;
compare a plurality of checksum values for equality, each checksum value being associated with a respective one of the storage container nodes, with a respective timestamp, and with a respective version of the designated data portion;
when it is determined that at least a first one of the plurality of checksum values does not match at least a second one of the plurality of checksum values, identify a designated one of the checksum values based on the timestamps; and
update the two or more storage container nodes to each include the designated data portion.
13 . The system recited in claim 12 , wherein identifying the designated checksum value comprises identifying the latest timestamp associated with a checksum value present on any storage container node.
14 . The system recited in claim 12 , wherein the designated storage container node is further configured to:
determining whether the system is configured to enforce stable data writes.
15 . The system recited in claim 14 , wherein identifying the designated checksum value comprises identifying the latest timestamp associated with a checksum value present on a designated number of the storage container nodes when it is determined that the system is configured to enforce stable writes.
16 . The system recited in claim 15 , wherein the designated number of the storage container nodes comprises a majority of the storage container nodes.
17 . The system recited in claim 12 , wherein the designated storage container node is further configured to:
receive a request for the designated data portion; and
transmit the designated data portion in response to the request.
18 . One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:
receiving a request to resynchronize a designated data portion stored on two or more storage container nodes, the two or more storage container nodes each configured to store the designated data portion;
comparing a plurality of checksum values for equality, each checksum value being associated with a respective one of the storage container nodes, with a respective timestamp, and with a respective version of the designated data portion;
when it is determined that at least a first one of the plurality of checksum values does not match at least a second one of the plurality of checksum values, identifying a designated one of the checksum values based on the timestamps; and
updating the two or more storage container nodes to each include the designated data portion.
19 . The one or more non-transitory computer readable media recited in claim 18 , wherein identifying the designated checksum value comprises identifying the latest timestamp associated with a checksum value present on any storage container node.
20 . The one or more non-transitory computer readable media recited in claim 18 , the method further comprising:
determining whether the system is configured to enforce stable data writes, wherein identifying the designated checksum value comprises identifying the latest timestamp associated with a checksum value present on a designated number of the storage container nodes when it is determined that the system is configured to enforce stable writes.