Cross-silo data storage and deduplication
In some aspects, a computing system may generate a content-defined tree. A content-defined tree may be a tree of cryptographic hashes where each leaf is a hash of a chunk (e.g., data chunk) of a data object, and each parent node (e.g., interior node) is the hash of a concatenation of the hashes of the parent's children nodes. To create parent nodes for the leaf nodes, a computing system may group leaf nodes together based on a rolling hash (e.g., a rolling hash of the hashes of the leaf nodes) satisfying a condition. Each parent node may include a hash that represents the concatenation of the hashes of the leaf nodes that fall under the corresponding parent node.
1 . A system for using content defined trees to efficiently index and deduplicate data stored in multiple databases, the system comprising:
one or more processors; and
a non-transitory, computer readable medium having instructions recorded thereon that, when executed by the one or more processors, cause operations comprising:
generating a first content defined tree corresponding to a legacy database, wherein the first content defined tree comprises a first set of parent nodes, each parent node of the first set of parent nodes corresponding to a set of hashes that have been determined using a rolling hash, and wherein each parent node comprises a hash of a concatenation of each hash in a corresponding set of hashes;
obtaining a second content defined tree corresponding to a content addressed storage (CAS) database, wherein the second content defined tree comprises a second set of parent nodes, each parent node in the second set of parent nodes comprising a concatenated hash corresponding to a set of leaf nodes; and
based on comparing the first content defined tree with the second content defined tree, removing a duplicate portion of data from the legacy database or the CAS database.
2 . The system of claim 1 , wherein the instructions, when executed by the one or more processors, cause operations further comprising:
obtaining a request for a file associated with the CAS database;
based on the request, retrieving a file-specific content defined tree comprising a file-specific set of parent nodes indicating each chunk of data used to recreate the file;
determining a first node by traversing the file-specific content defined tree;
comparing the first node with the first set of parent nodes of the first content defined tree and the second set of parent nodes of the second content defined tree;
based on a hash of the first node matching a hash of a node of the first content defined tree, obtaining a set of child nodes, wherein a first subset of the set of child nodes is obtained from the legacy database and a second subset of the set of child nodes is obtained from the CAS database;
retrieving, based on the set of child nodes, a set of file chunks; and
reconstructing the file based on the set of file chunks.
3 . The system of claim 1 , wherein removing a duplicate portion of data from the legacy database or the CAS database comprises:
performing a search on the first content defined tree and the second content defined tree;
based on the search, determining that a parent node of the first content defined tree comprises a first hash that matches a second hash of a parent node of the second content defined tree; and
based on the first hash matching the second hash, deleting data associated with the parent node of the second content defined tree.
4 . The system of claim 1 , wherein the instructions, when executed by the one or more processors, cause operations further comprising:
based on a request for a file, determining, based on the first content defined tree corresponding to the legacy database, a plurality of chunks;
sending the plurality of chunks to a user device; and
based on determining that a change has been made to the file, modifying the second content defined tree of the CAS database to include an additional parent node corresponding to the change that was made to the file.
5 . A method for using content defined trees to index and deduplicate data stored in multiple databases, the method comprising:
generating a first content defined tree corresponding to a first database, wherein the first content defined tree comprises a first set of parent nodes, each parent node of the first set of parent nodes corresponding to a set of hashes that have been determined using a rolling hash, and wherein each parent node comprises a hash of a concatenation of each hash in a corresponding set of hashes;
obtaining a second content defined tree corresponding to a second database; and
based on comparing the first content defined tree with the second content defined tree, removing a duplicate portion of data from the first database or the second database.
6 . The method of claim 5 , further comprising:
obtaining a request for a file associated with the second database;
based on the request, retrieving a file-specific content defined tree comprising a file-specific set of parent nodes indicating each chunk of data used to recreate the file;
determining a first node by traversing the file-specific content defined tree; and
determining a location of data associated with the file based on a comparison of the first node with the first content defined tree.
7 . The method of claim 6 , wherein determining a location of data associated with the file comprises:
comparing the first node with the first set of parent nodes of the first content defined tree and a second set of parent nodes of the second content defined tree;
based on a hash of the first node matching a hash of a node of the first content defined tree, obtaining a set of child nodes, wherein a first subset of the set of child nodes is obtained from the first database and a second subset of the set of child nodes is obtained from the second database; and
retrieving, based on the set of child nodes, a set of file chunks.
8 . The method of claim 7 , wherein each child node comprises a hash usable as a key to retrieve a location of a chunk of the file.
9 . The method of claim 5 , wherein removing a duplicate portion of data from the first database or the second database comprises:
performing a search on the first content defined tree and the second content defined tree;
based on the search, determining that a parent node of the first content defined tree comprises a first hash that matches a second hash of a parent node of the second content defined tree; and
based on the first hash matching the second hash, deleting data associated with the parent node of the second content defined tree.
10 . The method of claim 5 , further comprising:
generating a user interface comprising an indication of a node in the first content defined tree that matches a node in the second content defined tree; and
causing display of the user interface.
11 . The method of claim 5 , wherein the second content defined tree comprises a second set of parent nodes, each parent node in the second set of parent nodes comprising a concatenated hash corresponding to a set of leaf nodes.
12 . The method of claim 5 , further comprising:
based on a request for a file, determining, based on the first content defined tree corresponding to the first database, a plurality of chunks;
sending the plurality of chunks to a user device; and
based on determining that a change has been made to the file, modifying the second content defined tree of the second database to include an additional parent node corresponding to the change that was made to the file.
13 . A non-transitory, computer-readable medium comprising instructions that when executed by one or more processors, causes operations comprising:
generating a first content defined tree corresponding to a first database, wherein the first content defined tree comprises a first set of parent nodes, each parent node of the first set of parent nodes corresponding to a set of hashes that have been determined using a rolling hash, and wherein each parent node comprises a hash of a concatenation of each hash in a corresponding set of hashes;
obtaining a second content defined tree corresponding to a second database; and
based on comparing the first content defined tree with the second content defined tree, removing a duplicate portion of data from the first database or the second database.
14 . The non-transitory, computer-readable medium of claim 13 , further comprising:
obtaining a request for a file associated with the second database;
based on the request, retrieving a file-specific content defined tree comprising a file- specific set of parent nodes indicating each chunk of data used to recreate the file;
determining a first node by traversing the file-specific content defined tree; and
determining a location of data associated with the file based on a comparison of the first node with the first content defined tree.
15 . The non-transitory, computer-readable medium of claim 14 , wherein determining a location of data associated with the file comprises:
comparing the first node with the first set of parent nodes of the first content defined tree and a second set of parent nodes of the second content defined tree;
based on a hash of the first node matching a hash of a node of the first content defined tree, obtaining a set of child nodes, wherein a first subset of the set of child nodes is obtained from the first database and a second subset of the set of child nodes is obtained from the second database; and
retrieving, based on the set of child nodes, a set of file chunks.
16 . The non-transitory, computer-readable medium of claim 15 , wherein each child node comprises a hash usable as a key to retrieve a location of a chunk of the file.
17 . The non-transitory, computer-readable medium of claim 13 , wherein removing a duplicate portion of data from the first database or the second database comprises:
performing a search on the first content defined tree and the second content defined tree;
based on the search, determining that a parent node of the first content defined tree comprises a first hash that matches a second hash of a parent node of the second content defined tree; and
based on the first hash matching the second hash, deleting data associated with the parent node of the second content defined tree.
18 . The non-transitory, computer-readable medium of claim 13 , further comprising:
generating a user interface comprising an indication of a node in the first content defined tree that matches a node in the second content defined tree; and
causing display of the user interface.
19 . The non-transitory, computer-readable medium of claim 13 , wherein the second content defined tree comprises a second set of parent nodes, each parent node in the second set of parent nodes comprising a concatenated hash corresponding to a set of leaf nodes.
20 . The non-transitory, computer-readable medium of claim 13 , further comprising:
based on a request for a file, determining, based on the first content defined tree corresponding to the first database, a plurality of chunks;
sending the plurality of chunks to a user device; and
based on determining that a change has been made to the file, modifying the second content defined tree of the second database to include an additional parent node corresponding to the change that was made to the file.