Merkle tree forest for synchronizing data buckets of unlimited size in object storage systems
Embodiments extend using sparse Merkle trees for smart synchronization of S3 buckets by overcoming fixed size limitations through creating another Merkle tree when the fixed size limit of the first tree is exceeded, and creating yet another tree when the second tree is filled up, and so on as needed. The method maintains a list of trees, in which each tree can be synchronized separately by keeping a strict division to trees according to generation number. The generation is passed from a source site to a target site during replication operations. The tagging of the generation number also makes it easy and efficient to remove an older version of an element or deal with deleted elements. This allows efficient syncing between two data object buckets without a size limitation on number of elements in a bucket.
1. A method of synchronizing object data between a source site and a target site, comprising:
creating, in each of the source and target sites, an initial Merkle tree having a fixed size with each node having a hashed value of the metadata for the node and that of any children of that node;
receiving data to be stored in the initial Merkle tree until the fixed size is reached;
creating, upon reaching the fixed size, additional Merkle trees each of a respective fixed size in a sequence of successive additional Merkle trees as each additional Merkle tree receives data in excess of its respective fixed size;
associating a unique generation number with each of the initial Merkle tree and additional Merkle trees, as maintained in an index table;
determining, from the index table the existence of any missing additional Merkle trees that are in the source site but not in the target site; and
copying data of the missing additional Merkle trees from the source site to the target site using a Merkle tree synchronization, and wherein a new object of the received data is created in a POST operation by entering an element key and hash of the object value to a current Merkle tree, an existing data object is updated in a PUT operation by deleting its previous version from a previous Merkle tree in which the previous version resides, and an existing data object is deleted in a DELETE operation by fetching a corresponding generation tag for the existing data object and deleting it from the corresponding Merkle tree process.
2. The method of claim 1 wherein the object data comprises Amazon Simple Storage Service (S3) data.
3. The method of claim 1 wherein the Merkle tree is a sparse Merkle tree wherein a hash of an empty node is defined as zero, and includes nodes within the Merkle tree.
4. The method of claim 2 wherein the fixed size of the initial Merkle tree is of size M=c*n, wherein n is a maximum allowed number of elements in the bucket, and c is a single-digit integer constant.
5. The method of claim 4 wherein a size of each subsequent Merkle tree is the same size of the initial Merkle tree.
6. The method of claim 5 wherein a size of each subsequent Merkle tree increases relative to the size of the initial Merkle tree according to a defined sizing policy, and wherein the defined sizing policy comprises one of: increasing a subsequent Merkle tree size by a constant multiplier, or doubling a size of each subsequent Merkle tree or group of Merkle trees after the initial Merkle tree.
7. The method of claim 1 wherein the Merkle tree synchronization process comprises: recursively scanning child nodes of the missing additional Merkle trees to identify data blocks that have different hashes; and
sending data corresponding to the different hashes from the source node to the target node.
8. The method of claim 2 wherein the unique generation number is stored as object metadata for each S3 data object.
9. The method of claim 1 further comprising:
receiving, in the target site, replicated data from the source
sending the replicated data with an associated generation tag and object metadata to an appropriate Merkle tree on the target site if the replicated data object is already tagged;
creating a new Merkle tree of the sequence of successive additional Merkle trees if the data object is not already tagged; and
associating a new generation tag for the new Merkle tree.
10. A method of synchronizing object data in a data backup system, comprising:
maintaining a sequence of fixed-size Merkle trees for a source site and a target site;
receiving data to be replicated from the source site to the target site;
storing the received data in a current Merkle tree of the sequence of Merkle trees;
creating new Merkle trees as the received data exceeds the fixed size of the current Merkle tree, wherein each new Merkle tree is assigned a unique generation number; and
linking the unique generation number of each new Merkle tree to the corresponding new Merkle tree in an index table, and wherein a new object of the received data is created in a POST operation by entering an element key and hash of the object value to a current Merkle tree, an existing data object is updated in a PUT operation by deleting its previous version from a previous Merkle tree in which the previous version resides, and an existing data object is deleted in a DELETE operation by fetching a corresponding generation tag for the existing data object and deleting it from the corresponding Merkle tree.
11. The method of claim 10 wherein the object data comprises Amazon Simple Storage Service (S3) data.
12. The method of claim 11 wherein the Merkle tree is a sparse Merkle tree wherein a hash of an empty node is defined as zero, and includes nodes within the Merkle tree.
13. The method of claim 12 wherein a size of each subsequent Merkle tree is the same size of the initial Merkle tree.
14. The method of claim 13 wherein a size of each subsequent Merkle tree increases relative to the size of the initial Merkle tree according to a defined sizing policy, and wherein the defined sizing policy comprises one of: increasing a subsequent Merkle tree size by a constant multiplier, or doubling a size of each subsequent Merkle tree or group of Merkle trees after the initial Merkle tree.
15. The method of claim 14 wherein the unique generation number is stored as object metadata for each S3 data object.
16. The method of claim 10 wherein an initial Merkle tree of the sequence of fixed-size Merkle trees is denoted as Generation 1 , a second Merkle tree of the sequence of fixed-size Merkle trees is denoted as Generation 2 , a third Merkle tree of the sequence of fixed-size Merkle trees is denoted as Generation 3 , and a fourth Merkle tree of the sequence of fixed-size Merkle trees is denoted as Generation 4 .
17. The method of claim 16 wherein newly received data is input to a latest generation Merkle tree of the sequence of fixed-size Merkle trees.
18. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to execute a method of synchronizing object stores in a data backup system, comprising:
maintaining a sequence of fixed-size Merkle trees for a source site and a target site;
receiving data to be replicated from the source site to the target site;
storing the received data in a current Merkle tree of the sequence of Merkle trees;
creating new Merkle trees as the received data exceeds the fixed size of the current Merkle tree, wherein each new Merkle tree is assigned a unique generation number; and
linking the unique generation number of each new Merkle tree to the corresponding new Merkle tree in an index table, and wherein a new object of the received data is created in a POST operation by entering an element key and hash of the object value to a current Merkle tree, an existing data object is updated in a PUT operation by deleting its previous version from a previous Merkle tree in which the previous version resides, and an existing data object is deleted in a DELETE operation by fetching a corresponding generation tag for the existing data object and deleting it from the corresponding Merkle tree.