Web-scale distributed deduplication
Approaches for parallelized data deduplication. An instruction to perform data deduplication on a plurality of files is received. The plurality of files is organized into two or more work sets that each correspond to a subset of the plurality of files. Responsibility for performing each of said two or more work sets is assigned to a set of nodes in a cluster of nodes. The nodes may be physical nodes or virtual nodes. Each node in the set performs data deduplication on a different work set. In performing data deduplication, each node may store metadata describing where shared chunks of data are maintained in a distributed file system. The shared chunks of data are two or more sequences of bytes which appear in two or more of said plurality of files.
1. A method, comprising:
scanning a set of files to determine how to divide the set of files into a plurality of subsets of files;
assigning a corresponding subset of the plurality of subsets of files to each node of a plurality of nodes of a cluster;
providing the corresponding assigned subsets of files to each of the cluster nodes, wherein each of the cluster nodes is configured to:
perform, in parallel, deduplication with respect to the corresponding assigned subset of files, comprising to:
create, using a fingerprint algorithm, variable sized chunks of data associated with a file of the corresponding assigned subset of files;
create fingerprints of the variable sized chunks using a hash algorithm;
determine whether a fingerprint of the fingerprints already exists or is present in a parallel database; and
in response to a determination that the fingerprint does not already exist or is not present in the parallel database:
add a new entry to a database table, wherein the new entry identifies a current chunk associated with the fingerprint, a file offset in the file where the current chunk is stored, and the length of the current chunk;
update a lookup table with location information associated with the current chunk; and
update a block table to, for the file, indicate the fingerprint, offset, and length information; and
generate corresponding deduplication statistics associated with the corresponding assigned subset of files; and
assigning an additional subset of files to a node from which a notification that deduplication has been completed with respect to an assigned subset of files is received;
aggregating from each of the cluster nodes the corresponding deduplication statistics.
2. The method of claim 1 , wherein the set of files is associated with a directory or folder.
3. The method of claim 1 , further comprising dividing the set of files into the plurality of subsets of files.
4. The method of claim 3 , wherein dividing the set of files into the plurality of subsets of files comprises assigning one or more files included in the set of files to a subset of files of the plurality of subsets of files until a threshold size of files is assigned to the subset of files.
5. The method of claim 3 , wherein a node of the plurality of cluster nodes is assigned an additional subset of files after each of the other cluster nodes is assigned an initial subset of files.
6. The method of claim 1 , wherein the corresponding subset of the plurality of subsets of files is assigned to each cluster node of the plurality of cluster nodes based on available bandwidth or processing power.
7. The method of claim 1 , wherein a node of the plurality of cluster nodes is comprised of a plurality of compute containers.
8. The method of claim 7 , wherein the node assigns a received subset of files one of the plurality of compute containers.
9. The method of claim 7 , further comprising receiving the notification that deduplication has been completed with respect to the assigned subset of files.
10. A computer program product, the computer program product being embodied in non-transitory computer readable medium and comprising instructions for:
scanning a set of files to determine how to divide the set of files into a plurality of subsets of files;
assigning a corresponding subset of the plurality of subsets of files to each node of a plurality of nodes of a cluster;
providing the corresponding assigned subsets of files to each of the cluster nodes, wherein each of the cluster nodes is configured to:
perform, in parallel, deduplication with respect to the corresponding assigned subset of files, comprising to:
create, using a fingerprint algorithm, variable sized chunks of data associated with a file of the corresponding assigned subset of files;
create fingerprints of the variable sized chunks using a hash algorithm;
determine whether a fingerprint of the fingerprints already exists or is present in a parallel database; and
in response to a determination that the fingerprint does not already exist or is not present in the parallel database:
add a new entry to a database table, wherein the new entry identifies a current chunk associated with the fingerprint, a file offset in the file where the current chunk is stored, and the length of the current chunk;
update a lookup table with location information associated with the current chunk; and
update a block table to, for the file, indicate the fingerprint, offset, and length information; and
generate corresponding deduplication statistics associated with the corresponding assigned subset of files; and
assigning an additional subset of files to a node from which a notification that deduplication has been completed with respect to an assigned subset of files is received;
aggregating from each of the cluster nodes the corresponding deduplication statistics.
11. The computer program product of claim 10 , further comprising instructions for dividing the set of files into the plurality of subsets of files.
12. The computer program product of claim 11 , wherein dividing the set of files into the plurality of subsets of files comprises assigning one or more files included in the set of files to a subset of files of the plurality of subsets of files until a threshold size of files is assigned to the subset of files.
13. The computer program product of claim 11 , wherein a node of the plurality of cluster nodes is assigned an additional subset of files after each of the other cluster nodes is assigned an initial subset of files.
14. The computer program product of claim 10 , wherein a node of the plurality of cluster nodes is comprised of a plurality of compute containers.
15. The computer program product of claim 14 , wherein the node assigns a received subset of files one of the plurality of compute containers.
16. The computer program product of claim 14 , further comprising instructions for receiving the notification that deduplication has been completed with respect to the assigned subset of files.
17. The computer program product of claim 10 , wherein the set of files is associated with a directory or folder.
18. A system, comprising:
a processor; and
a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
scan a set of files to determine how to divide the set of files into a plurality of subsets of files;
assign a corresponding subset of the plurality of subsets of files to each node of a plurality of nodes of a cluster;
provide the corresponding assigned subsets of files to each of the cluster nodes, wherein each of the cluster nodes is configured to:
perform, in parallel, deduplication with respect to the corresponding assigned subset of files, comprising to:
create, using a fingerprint algorithm, variable sized chunks of data associated with a file of the corresponding assigned subset of files;
create fingerprints of the variable sized chunks using a hash algorithm;
determine whether a fingerprint of the fingerprints already exists or is present in a parallel database; and
in response to a determination that the fingerprint does not already exist or is not present in the parallel database:
add a new entry to a database table, wherein the new entry identifies a current chunk associated with the fingerprint, a file offset in the file where the current chunk is stored, and the length of the current chunk;
update a lookup table with location information associated with the current chunk; and
update a block table to, for the file, indicate the fingerprint, offset, and length information; and
generate corresponding deduplication statistics associated with the corresponding assigned subset of files; and
assign an additional subset of files to a node from which a notification that deduplication has been completed with respect to an assigned subset of files is received;
aggregate from each of the cluster nodes the corresponding deduplication statistics.