Method and system for I/O parallel distributed garbage collection of a deduplicated datasets
Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a method comprising creating a first set of temporary files associated with a range of fingerprints for data within data files associated with a directory tree structure; creating a second set of temporary files associated with a range of fingerprints of storage segments stored on one or more deduplicated storage containers; sorting the fingerprints in each temporary file using distributed out of core sorting across each node in the set of multiple computing device nodes to generate a first set of sorted files and a second set of sorted files; determining an intersection of the fingerprints in the first set of sorted files and the second set of sorted files; and generating a garbage collection recipe for each of the one or more deduplicated storage containers.
1. A method for distributed garbage collection of deduplicated datasets, the method comprising:
creating a first set of temporary files, and storing the first set of temporary files across a plurality of nodes in a set of multiple computing device nodes, where each temporary file of the first set of temporary files stores a range of fingerprints for data within data files associated with a directory tree structure;
creating a second set of temporary files, and storing the second set of temporary files across a second plurality of nodes in the set of multiple computing device nodes, where each temporary file of the second set of temporary files stores a range of fingerprints of storage segments stored on one or more deduplicated storage containers;
sorting the fingerprints stored in each temporary file using distributed out of core sorting across each node in the set of multiple computing device nodes, wherein each temporary file of the first set and the second set are sorted in parallel at each corresponding node where the temporary file is stored, and sorting information including the temporary files of the first set of temporary files and the second set of temporary files is exchanged between the nodes to generate a first set of sorted files and a second set of sorted files and sorting the fingerprints includes transferring a first temporary file storing a first range of fingerprints to a first node associated with the first range of fingerprints, receiving a second temporary file associated with a second range of fingerprints, and storing the second temporary file in local storage;
determining an intersection of the fingerprints in the first set of sorted files and the second set of sorted files; and
generating a garbage collection recipe for each of the one or more deduplicated storage containers based on the intersection of the fingerprints.
2. The method as in claim 1 , additionally comprising sorting fingerprints in locally stored temporary files to generate the first set of sorted files and the second set of sorted files, the first set of sorted files including the fingerprints for data within data files stored in a directory tree structure and the second set of sorted files including the fingerprints of the storage segments stored on the one or more deduplicated storage containers.
3. The method as in claim 2 , additionally comprising finding the intersection between the first set of sorted files and the second set of sorted files, the intersection representing the fingerprints of data stored in the storage containers which are referenced by a file in the directory tree structure.
4. The method as in claim 3 , additionally comprising finding the intersection by sequentially walking the first set of sorted files and the second set of sorted files to identify fingerprints in common.
5. The method as in claim 3 , wherein generating the garbage collection recipe includes sorting the intersection of fingerprints based on a container identifier associated with each fingerprint, each container identifier identifying one of the one or more deduplicated storage containers.
6. The method as in claim 5 , wherein sorting the intersection of fingerprints includes:
creating a third set of temporary files including the intersection of fingerprints, each temporary file in the third set of temporary files stored on a node in the set of multiple computing device nodes;
sorting the intersection of fingerprints using distributed out of core sorting across each node in the set of multiple computing device nodes; and
for each node in the set of multiple computing device nodes, storing a fingerprint into a file based on the container identifier associated with the fingerprint.
7. The method as in claim 5 , wherein sorting the intersection of fingerprints using distributed out of core sorting additionally includes, for each node in the set of multiple computing device nodes:
assigning a range of container identifiers to the node;
transferring a set of fingerprints associated with a container identifier to the node assigned to that container identifier;
receiving the set of fingerprints associated with the container identifier for the node; and
storing the set of fingerprints in a file to generate the garbage collection recipe for a container identifier.
8. The method as in claim 1 , wherein determining the intersection of the fingerprints in the first set of temporary files and the second set of temporary files includes determining a set of revived storage segments stored on the one or more de duplicated storage containers, the revived storage segments associated with one or more data files stored in the directory tree structure after creating the first set of temporary files.
9. The method as in claim 1 , additionally comprising:
via one or more nodes in the set of multiple computing device nodes, reading the garbage collection recipe associated with one of the one or more deduplicated storage containers;
copying the storage segments identified in the garbage collection recipe from the one or more deduplicated storage containers to a new storage container; and
deleting the one or more deduplicated storage containers.
10. The method as in claim 9 , additionally comprising copying the storage segments identified in the garbage collection recipe in parallel for multiple deduplicated storage containers.
11. A system to perform distributed garbage collection of deduplicated datasets, the system comprising:
one or more deduplicated storage containers;
a processor; and
a memory coupled to the processor for storing instructions, which when executed from the memory, cause the processor to perform operations including:
creating a first set of temporary files, and storing the first set of temporary files across a plurality of nodes in a set of multiple computing device nodes, where each temporary file of the first set of temporary files stores a range of fingerprints for data within data files associated with a directory tree structure;
creating a second set of temporary files, and storing the second set of temporary files across a second plurality of nodes in the set of multiple computing device nodes, where each temporary file of the second set of temporary files stores a range of fingerprints of storage segments stored on the one or more deduplicated storage containers;
sorting the fingerprints stored in each temporary file using distributed out of core sorting across each node in the set of multiple computing device nodes, wherein each temporary file of the first set and the second set are sorted in parallel at each corresponding node where the temporary file is stored, and sorting information including the temporary files of the first set of temporary files and the second set of temporary files is exchanged between the nodes to generate a first set of sorted files and a second set of sorted files, and sorting the fingerprints includes transferring a first temporary file storing a first range of fingerprints to a first node associated with the first range of fingerprints, receiving a second temporary file associated with a second range of fingerprints, and storing the second temporary file in local storage;
determining an intersection of the fingerprints in the first set of sorted files and the second set of sorted files; and
generating a garbage collection recipe for each of the one or more deduplicated storage containers based on the intersection of the fingerprints.
12. The system as in claim 11 , additionally comprising sorting fingerprints in locally stored temporary files to generate the first set of sorted files and the second set of sorted files, the first set of sorted files including the fingerprints for data within data files stored in a directory tree structure and the second set of sorted files including the fingerprints of the storage segments stored on the one or more deduplicated storage containers.
13. The system as in claim 12 , additionally comprising finding the intersection between the first set of sorted files and the second set of sorted files, the intersection representing the fingerprints of data stored in the storage containers which are referenced by a file in the directory tree structure.
14. The system as in claim 13 , additionally comprising finding the intersection by sequentially walking the first set of sorted files and the second set of sorted files to identify fingerprints in common.
15. A non-transitory machine-readable medium storing instructions which, when executed by one or more processors of a data processing system, cause the data processing system to perform operations including:
creating a first set of temporary files, and storing the first set of temporary files across a plurality of nodes in a set of multiple computing device nodes, where each temporary file of the first set of temporary files stores a range of fingerprints for data within data files associated with a directory tree structure;
creating a second set of temporary files, and storing the second set of temporary files across a second plurality of nodes in the set of multiple computing device nodes, where each temporary file of the second set of temporary files stores a range of fingerprints of storage segments stored on one or more deduplicated storage containers;
sorting the fingerprints stored in each temporary file using distributed out of core sorting across each node in the set of multiple computing device nodes, wherein each temporary file of the first set and the second set are sorted in parallel at each corresponding node where the temporary file is stored, and sorting information including the temporary files of the first set of temporary files and the second set of temporary files is exchanged between the nodes to generate a first set of sorted files and a second set of sorted files, and sorting the fingerprints includes transferring a first temporary file storing a first range of fingerprints to a first node associated with the first range of fingerprints, receiving a second temporary file associated with a second range of fingerprints, and storing the second temporary file in local storage;
determining an intersection of the fingerprints in the first set of sorted files and the second set of sorted files; and
generating a garbage collection recipe for each of the one or more deduplicated storage containers based on the intersection of the fingerprints.
16. The non-transitory machine-readable medium as in claim 15 , additionally comprising sorting fingerprints in locally stored temporary files to generate the first set of sorted files and the second set of sorted files, the first set of sorted files including the fingerprints for data within data files stored in a directory tree structure and the second set of sorted files including the fingerprints of the storage segments stored on the one or more deduplicated storage containers.
17. The non-transitory machine-readable medium as in claim 16 , additionally comprising finding the intersection between the first set of sorted files and the second set of sorted files, the intersection representing the fingerprints of data stored in the storage containers which are referenced by a file in the directory tree structure.