Using sparse merkle trees for smart synchronization of S3
One example method, which may be performed in connection with an object store, includes receiving a key of a key-value pair, correlating the key to a location in a base of a Merkle tree, inserting the key at the location, hashing the value associated with the key to produce a data hash, and inserting the data hash in the Merkle tree. The Merkle tree may then be checked for consistency, and synchronized with another Merkle tree. The Merkle tree may be of a fixed size, and insertion of the key in the Merkle tree does not change the location of any keys existing in the Merkle tree prior to insertion of the new key.
1. A method, comprising:
receiving a key of a key-value pair;
correlating the key to a location in a base of a Merkle tree based on a hash of the key;
after the correlating is performed, inserting the hash of the key at the location;
hashing a value associated with the key to produce a data hash;
inserting the data hash in the Merkle tree; and
updating the Merkle tree,
wherein updating the Merkle tree comprises calculating any hashes, which include a root hash of the Merkle tree, affected by insertion of the data hash, and
wherein insertion of the hash at the location does not change a location of a hash of another key present in the base of the Merkle tree when the hash was inserted.
2. The method as recited in claim 1 , wherein correlating the key to a location comprises hashing the key and determining the location based on the hash of the key.
3. The method as recited in claim 1 , wherein the Merkle tree is a sparse Merkle tree.
4. The method as recited in claim 1 , wherein the value of the key-value pair comprises an object.
5. The method as recited in claim 1 , wherein the Merkle tree comprises a map of objects of a dataset.
6. The method as recited in claim 1 , wherein the method is performed in response to an IO written to a dataset.
7. The method as recited in claim 1 , wherein the Merkle tree corresponds to a dataset of unordered objects.
8. The method as recited in claim 1 , wherein the Merkle tree is a non-binary tree.
9. The method as recited in claim 1 , further comprising performing a consistency check of the Merkle tree, and synchronizing the Merkle tree with another Merkle tree.
10. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving a key of a key-value pair;
correlating the key to a location in a base of a Merkle tree based on a hash of the key;
after the correlating is performed, inserting the hash of the key at the location;
hashing the value associated with the key to produce a data hash;
inserting the data hash in the Merkle tree; and
updating the Merkle tree,
wherein updating the Merkle tree comprises calculating any hashes, which includes a root hash of the Merkle tree, affected by insertion of the data hash, and
wherein insertion of the hash at the location does not change a location of a hash of another key present in the base of the Merkle tree when the hash was inserted.
11. The non-transitory storage medium as recited in claim 10 , wherein correlating the key to a location comprises hashing the key and determining the location based on the hash of the key.
12. The non-transitory storage medium as recited in claim 10 , wherein the Merkle tree is a sparse Merkle tree.
13. The non-transitory storage medium as recited in claim 10 , wherein the value of the key-value pair comprises an object.
14. The non-transitory storage medium as recited in claim 10 , wherein the Merkle tree comprises a map of objects of a dataset.
15. The non-transitory storage medium as recited in claim 10 , wherein the operations are performed in response to an IO written to a dataset.
16. The non-transitory storage medium as recited in claim 10 , wherein the Merkle tree corresponds to a dataset of unordered objects.
17. The non-transitory storage medium as recited in claim 10 , wherein the Merkle tree is a non-binary tree.
18. The non-transitory storage medium as recited in claim 10 , wherein the operations further comprise performing a consistency check of the Merkle tree, and synchronizing the Merkle tree with another Merkle tree.