Method and system for reducing memory used in embedded DDRs by using spare drives for OOC GC
Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a data storage system comprising an array of redundant storage devices including a first set of storage devices to be configured as live storage devices and a second set of storage devices to be configured as spare storage devices, a spare storage device to be enabled in event of a failure of a live storage device; and a set of processing devices coupled to the array of redundant storage devices, the set of processing devices to execute logic to enable data replication and deduplication for the array of redundant storage devices and perform distributed deduplication garbage collection on the first set of storage devices using one or more devices in the second set of storage devices as temporary storage.
1. A data storage system comprising:
an array of redundant storage devices including a first set of storage devices to be configured as live storage devices and a second set of storage devices to be configured as spare storage devices, a spare storage device to be enabled in event of a failure of a live storage device; and
a set of processing devices coupled to the array of redundant storage devices, the set of processing devices to execute logic to enable data replication and deduplication for the array of redundant storage devices and perform distributed deduplication garbage collection on the first set of storage devices using one or more devices in the second set of storage devices as temporary storage, wherein the set of processing devices, to perform distributed deduplication garbage collection, is configured to:
create a first set of temporary files, each temporary file in the first set of temporary files stored on one or more of the second set of storage devices and associated with a range of fingerprints for data within data files stored in a directory tree structure in the first set of storage devices; and
create a second set of temporary files, each temporary file in the second set of temporary files stored on one or more of the second set of storage devices and associated with a range of fingerprints of storage segments stored on one or more deduplicated storage containers on the first set of storage devices.
2. The data storage system as in claim 1 , wherein the set of processing devices are virtual processing devices managed by a virtual machine manager.
3. The data storage system as in claim 1 , wherein the set of processing devices are configured to create the first set of temporary files and in parallel with creation of the second set of temporary files.
4. The data storage system as in claim 1 , wherein the set of processing devices are further configured to:
sort the fingerprints in each temporary file via a distributed out-of-core sort across each processing device in the set of processing devices;
determine an intersection of the fingerprints in the first set of temporary files and the second set of temporary files; and
generate a garbage collection recipe for each of the one or more deduplicated storage containers.
5. The data storage system as in claim 4 , wherein determining an intersection of the fingerprints in the first set of temporary files and the second set of temporary files comprises determining a set of revived storage segments stored on the one or more deduplicated storage containers, the revived storage segments associated with one or more data files stored in the directory tree structure, the association created after creation of the first set of temporary files.
6. The data storage system as in claim 4 , wherein generating the garbage collection recipe comprises sorting an intersection of fingerprints from the two sets of temporary files based on a container identifier associated with each fingerprint, each container identifier to identify one of the one or more deduplicated storage containers.
7. The data storage system as in claim 6 , wherein sorting the intersection of fingerprints comprises:
creating a third set of temporary files including the intersection of fingerprints, each temporary file in the third set of temporary files to be stored on a storage device in the second set of storage devices and associated with one processing device in the set of processing devices; and
sorting the intersection of fingerprints via a distributed out of core sort across each processing devices, wherein each processing device is to store a fingerprint into a file on the second set of storage devices based on the container identifier associated with the fingerprint.
8. The data storage system as in claim 6 , wherein to sort the intersection of fingerprints, each processing device in the set of processing devices is further configured to:
transfer a set of fingerprints associated with a container identifier to the processing device assigned to that container identifier;
receive the set of fingerprints associated with the container identifier for the processing device; and
store the set of fingerprints in a file to generate the garbage collection recipe for a container identifier.
9. The data storage system as in claim 6 , wherein one or more processing devices in the set of processing devices are configured to:
read the garbage collection recipe associated with one of the one or more deduplicated storage containers;
copy the storage segments identified in the garbage collection recipe from the one or more deduplicated storage containers to a new storage container; and
delete the one or more deduplicated storage containers.
10. The data storage system as in claim 9 , wherein the one or more processing devices are further configured to copy the storage segments identified in the garbage collection recipe in parallel for multiple deduplicated storage containers.
11. A computer implemented method comprising:
configuring an array of redundant storage devices including a first set of storage devices and a second set of storage devices, the second set of storage devices configured as spare storage devices; and
performing distributed deduplication garbage collection on deduplicated storage containers on the first set of storage devices using one or more devices in the second set of storage devices as temporary storage, wherein performing distributed deduplication garbage collection comprises
creating a first set of temporary files, each temporary file in the first set of temporary files stored on one or more of the second set of storage devices and associated with a range of fingerprints for data within data files stored in a directory tree structure in the first set of storage devices; and
creating a second set of temporary files, each temporary file in the second set of temporary files stored on one or more of the second set of storage devices and associated with a range of fingerprints of storage segments stored on one or more deduplicated storage containers on the first set of storage devices.
12. The computer implemented method as in claim 11 , further comprising creating the first set of temporary files and in parallel with creating the second set of temporary files.
13. The computer implemented method as in claim 11 , further comprising:
sorting the fingerprints in each temporary file via a distributed out-of-core sort across each processing device in a set of processing devices;
determine an intersection of the fingerprints in the first set of temporary files and the second set of temporary files; and
generate a garbage collection recipe for each of the one or more deduplicated storage containers.
14. The computer implemented method as in claim 13 , wherein determining an intersection of the fingerprints in the first set of temporary files and the second set of temporary files includes determining a set of revived storage segments stored on the one or more deduplicated storage containers, the revived storage segments associated with one or more data files stored in the directory tree structure, the association created after creation of the first set of temporary files.
15. The computer implemented method as in claim 14 , further comprising aborting the distributed deduplication garbage collection in response to losing access to one or more devices in the second set of storage devices due to a failure of devices in the first set of storage devices.
16. A non-transitory machine-readable medium storing instructions which, when executed by one or more processors of a data processing system, cause the data processing system to perform operations including:
configuring an array of redundant storage devices including a first set of storage devices and a second set of storage devices, the second set of storage devices configured as spare storage devices;
performing distributed deduplication garbage collection on deduplicated storage containers on the first set of storage devices using one or more devices in the second set of storage devices as temporary storage, wherein performing distributed deduplication garbage collection comprises
creating a first set of temporary files, each temporary file in the first set of temporary files stored on one or more of the second set of storage devices and associated with a range of fingerprints for data within data files stored in a directory tree structure in the first set of storage devices, and
creating a second set of temporary files, each temporary file in the second set of temporary files stored on one or more of the second set of storage devices and associated with a range of fingerprints of storage segments stored on one or more deduplicated storage containers on the first set of storage devices.
17. The non-transitory machine-readable medium as in claim 16 , wherein the operations further comprise creating the first set of temporary files and in parallel with creating the second set of temporary files.
18. The non-transitory machine-readable medium as in claim 16 , wherein the operations further comprise:
sorting the fingerprints in each temporary file via a distributed out-of-core sort across each processing device in a set of processing devices;
determine an intersection of the fingerprints in the first set of temporary files and the second set of temporary files; and
generate a garbage collection recipe for each of the one or more deduplicated storage containers.
19. The non-transitory machine-readable medium as in claim 18 , wherein determining an intersection of the fingerprints in the first set of temporary files and the second set of temporary files includes determining a set of revived storage segments stored on the one or more deduplicated storage containers, the revived storage segments associated with one or more data files stored in the directory tree structure, the association created after creation of the first set of temporary files.
20. The non-transitory machine-readable medium as in claim 19 , wherein the operations further comprise aborting the distributed deduplication garbage collection in response to losing access to one or more devices in the second set of storage devices due to a failure of devices in the first set of storage devices.