Data indexing and deduplication using content-defined trees
In some aspects, a computing system may generate a content-defined tree. A content-defined tree may be a tree of cryptographic hashes where each leaf is a hash of a chunk (e.g., data chunk) of a data object, and each parent node (e.g., interior node) is the hash of a concatenation of the hashes of the parent's children nodes. To create parent nodes for the leaf nodes, a computing system may group leaf nodes together based on a rolling hash (e.g., a rolling hash of the hashes of the leaf nodes) satisfying a condition. Each parent node may include a hash that represents the concatenation of the hashes of the leaf nodes that fall under the corresponding parent node.
1 . A system for storing large data objects in a format that can be efficiently modified through use of content-defined trees, the system comprising:
one or more processors; and
a non-transitory, computer-readable medium having instructions recorded thereon that, when executed by the one or more processors, cause operations comprising:
obtaining a data object comprising a string of bytes;
dividing the string of bytes into a set of chunks, each chunk in the set of chunks having a boundary, wherein each boundary is determined based on a first rolling hash satisfying a first condition and each boundary defines a size of a corresponding chunk;
generating a content-defined tree by:
generating a set of hashes comprising a cryptographic hash for each chunk of the set of chunks, wherein the set of hashes form a first tier of the content-defined tree; and
generating a set of parent nodes by grouping each hash of the set of hashes based on a second rolling hash satisfying a second condition, and by hashing a concatenation of each resulting group of hashes, wherein the set of parent nodes form a second tier of the content-defined tree, wherein a first parent node of the set of parent nodes comprises a hash that is usable as a key to retrieve each hash in a group of hashes that corresponds to the first parent node; and
storing a portion of the content-defined tree in a database.
2 . A computer-implemented method for storing large data objects in a format that can be efficiently modified through use of content-defined trees, the method comprising executing, by one or more processors, operations comprising:
obtaining a data object comprising a string of bytes;
dividing the string of bytes into a set of chunks, each chunk in the set of chunks having a boundary, wherein each boundary is determined based on a first rolling hash satisfying a first condition and each boundary defines a size of a corresponding chunk;
generating a content-defined tree by:
generating a set of hashes comprising a cryptographic hash for each chunk of the set of chunks, wherein the set of hashes form a first tier of the content-defined tree; and
generating a set of parent nodes by grouping each hash of the set of hashes based on a second rolling hash satisfying a second condition, and by hashing a concatenation of each resulting group of hashes, wherein the set of parent nodes form a second tier of the content-defined tree; and
storing a portion of the content-defined tree in a database.
3 . The computer-implemented method of claim 2 , further comprising:
generating a root node by merging each node in the set of parent nodes by, based on applying the second rolling hash and the second condition to the set of parent nodes, determining that each parent node in the set of parent nodes should be combined into one group; and
generating a hash of a concatenation of hashes corresponding to the set of parent nodes.
4 . The computer-implemented method of claim 2 , wherein the second condition is configured to provide an average group size of four.
5 . The computer-implemented method of claim 2 , wherein the second condition is configured to force a number of hashes in a group of hashes to be between two and eight.
6 . The computer-implemented method of claim 2 , wherein generating a set of parent nodes further comprises hashing a message authentication code with a corresponding concatenation of each resulting group of hashes.
7 . The computer-implemented method of claim 2 , wherein the data object is stored in a data repository with a set of data objects, the method further comprising executing, by one or more processors, operations comprising:
generating, based on the data repository, a metadata store comprising a directory layout and metadata of the data repository;
generating a byte stream comprising a concatenation of all bytes of all data object in the data repository, wherein the concatenation is sorted in hash order; and
generating a second content-defined tree based on the byte stream.
8 . The computer-implemented method of claim 7 , wherein generating the second content-defined tree comprises inserting a chunk boundary at an end of each data object in the data repository.
9 . The computer-implemented method of claim 7 , wherein the data repository corresponds to a dataset for training a machine learning model, the method further comprising executing, by one or more processors, operations comprising:
designating a first portion of the set of parent nodes as a training dataset and a second portion of the set of parent nodes as a testing dataset; and
training the machine learning model using the training dataset and the testing dataset.
10 . The computer-implemented method of claim 2 , wherein a first parent node of the set of parent nodes comprises a hash that is usable as a key to retrieve each hash in a group of hashes that corresponds to the first parent node.
11 . The computer-implemented method of claim 2 , further comprising executing, by one or more processors, operations comprising:
determining, based on a comparison of the content-defined tree with a second content-defined tree, that the data object has been modified; and
based on the data object having been modified, updating a hash of a parent node of the set of parent nodes indicating a modification.
12 . A non-transitory, computer-readable medium comprising instructions that when executed by one or more processors, causes operations comprising:
obtaining a data object comprising a string of bytes;
dividing the string of bytes into a set of chunks, each chunk in the set of chunks having a boundary, wherein each boundary is determined based on a first rolling hash satisfying a first condition and each boundary defines a size of a corresponding chunk;
generating a content-defined tree by:
generating a set of hashes comprising a cryptographic hash for each chunk of the set of chunks, wherein the set of hashes form a first tier of the content-defined tree; and
generating a set of parent nodes by grouping each hash of the set of hashes based on a second rolling hash satisfying a second condition, and by hashing a concatenation of each resulting group of hashes, wherein the set of parent nodes form a second tier of the content-defined tree; and
storing a portion of the content-defined tree in a database.
13 . The non-transitory, computer-readable medium of claim 12 further storing instructions which, when executed by the one or more processors, cause operations comprising:
generating a rood node by merging each node int eh set of parent nodes by, based on applying the second rolling hash and the second condition to the set of parent nodes, determining that each parent node in the set of parent nodes should be combined into one group; and
generating a hash of a concatenation of hashes corresponding to the set of parent nodes.
14 . The non-transitory, computer-readable medium of claim 12 , wherein the second condition is configured to provide an average group size of four.
15 . The non-transitory, computer-readable medium of claim 12 , wherein the second condition is configured to force a number of hashes in a group of hashes to be between two and eight.
16 . The non-transitory, computer-readable medium of claim 12 , wherein generating a set of parent nodes further comprises hashing a message authentication code with a corresponding concatenation of each resulting group of hashes.
17 . The non-transitory, computer-readable medium of claim 12 , wherein the data object is stored in a data repository with a set of data objects, the operations further comprising:
generating, based on the data repository, a metadata store comprising a directory layout and metadata of the data repository;
generating a byte stream comprising a concatenation of all bytes of all data object in the data repository, wherein the concatenation is sorted in hash order; and
generating a second content-defined tree based on the byte stream.
18 . The non-transitory, computer-readable medium of claim 17 , wherein generating the second content-defined tree comprises inserting a chunk boundary at an end of each data object in the data repository.
19 . The non-transitory, computer-readable medium of claim 17 , wherein the data repository corresponds to a dataset for training a machine learning model, the operations further comprising:
designating a first portion of the set of parent nodes as a training dataset and a second portion of the set of parent nodes as a testing dataset; and
training the machine learning model using the training dataset and the testing dataset.
20 . The non-transitory, computer-readable medium of claim 17 , wherein a first parent node of the set of parent nodes comprises a hash that is usable as a key to retrieve each hash in a group of hashes that corresponds to the first parent node.