Scalable garbage collection for deduplicated storage
Systems and methods for cleaning a storage system. A deduplicated storage system is cleaned by identifying structures that include dead or unreferenced segments. This includes processing recipes to identify the segments that are no longer part of a live object recipe. Then, the dead segments are removed. This is accomplished by copying forward the live segments and then deleting, as a whole, the structure that included the dead segments.
1. A method comprising:
identifying impacted similarity groups that are impacted by a garbage collection operation, wherein the impacted similarity groups include segments associated with deleted objects and segments associated with live objects;
identifying sub-groups, of each of the impacted similarity groups, impacted by the garbage collection operation;
write locking at least the impacted sub-groups of the impacted similarity groups in order to support a normal operation and the garbage collection operation concurrently in the impacted similarity groups;
directing normal operations to a highest numbered sub-group of each of the impacted similarity group, the normal operations including a write operation;
determining sizes of the impacted similarity groups individually and determining a size of all the impacted similarity groups;
determining a number of workers to perform the garbage collection operation that cleans the deleted objects stored in the storage system from objects stored in the storage system based on the sizes of the impacted similarity groups individually and a size of all the impacted similarity groups;
assigning each of the workers a range of the impacted similarity groups based on the sizes impacted similarity groups and the size of all the impacted similarity groups;
removing the segments associated with the deleted objects from the impacted sub-groups; and
updating the impacted similarity groups to reflect that the segments associated with the deleted objects have been removed.
2. The method of claim 1 , further comprising initiating a controller to control the garbage collection operation, wherein the controller controls the of workers.
3. The method of claim 2 , further comprising estimating, by the controller, the number of workers needed to perform the garbage collection operation.
4. The method of claim 3 , further estimating the number of workers by accounting for one or more factors that include an environment variable setting the number of workers, a memory allocation of each worker, a number and/or size of similarity groups impacted by the garbage collection process, Input/Output operations, or throughput.
5. The method of claim 1 , wherein removing the segments associated with the deleted objects includes copying the segments associated with the live objects in the impacted sub-groups to new sub-groups such that the new sub-groups do not store the segments that are no longer referenced by live objects, wherein the new sub-groups are included in the impacted similarity groups.
6. The method of claim 5 , further comprising deleting the impacted sub-groups after the new sub-groups are completed.
7. The method of claim 6 , further comprising retrieving records corresponding to the deleted objects, wherein the records identify at least object recipes for the impacted similarity groups and subgroups and wherein the impacted similarity groups and the impacted sub-groups are identified from slice recipes associated with the object recipes.
8. The method of claim 1 , further comprising:
identifying a highest numbered sub-group of a particular impacted similarity group, the highest numbered sub-group being marked for cleaning; and
adding a new sub-group to the particular similarity group, wherein the normal operation is directed to the new sub-group and no incoming write operations are directed to the impacted sub-groups that are locked.
9. The method of claim 8 , further comprising:
deduplicating one or more objects written to an object storage with respect to sub-groups that are not write locked in the impacted similarity groups, the one or more objects being written to an object storage during the garbage collection operation.
10. The method of claim 1 , wherein each of the subgroups in each of the impacted similarity groups is an independent object.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
identifying impacted similarity groups that are impacted by a garbage collection operation, wherein the impacted similarity groups include segments associated with deleted objects and segments associated with live objects;
identifying sub-groups, of each of the impacted similarity groups, impacted by the garbage collection operation;
write locking at least the impacted sub-groups of the impacted similarity groups in order to support a normal operation and the garbage collection operation concurrently in the impacted similarity groups;
directing normal operations to a highest numbered sub-group of each of the impacted similarity group, the normal operations including a write operation;
determining sizes of the impacted similarity groups individually and determining a size of all the impacted similarity groups;
determining a number of workers to perform the garbage collection operation that cleans the deleted objects stored in the storage system from objects stored in the storage system based on the sizes of the impacted similarity groups individually and a size of all the impacted similarity groups;
assigning each of the workers a range of the impacted similarity groups based on the sizes impacted similarity groups and the size of all the impacted similarity groups:
removing the segments associated with the deleted objects from the impacted sub-groups; and
updating the impacted similarity groups to reflect that the segments associated with the deleted objects have been removed.
12. The non-transitory storage medium of claim 11 , further comprising:
initiating a controller to control the garbage collection operation, wherein the controller controls the workers; and
estimating, by the controller, the number of workers needed to perform the garbage collection operation.
13. The non-transitory storage medium of claim 12 , further estimating the number of workers by accounting for one or more factors that include an environment variable setting the number of workers, a memory allocation of each worker, a number and/or size of similarity groups impacted by the garbage collection process, Input/Output operations, or throughput.
14. The non-transitory storage medium of claim 12 , wherein removing the segments associated with the deleted objects includes copying the segments associated with the live objects in the impacted sub-groups to new sub-groups such that the new sub-groups do not store the segments that are no longer referenced by live objects, wherein the new sub-groups are included in the impacted similarity groups.
15. The non-transitory storage medium of claim 14 , further comprising deleting the impacted sub-groups after the new sub-groups are completed, wherein each of the subgroups in each of the impacted similarity groups including the new sub-groups is an independent object.
16. The non-transitory storage medium of claim 15 , further comprising retrieving records corresponding to the deleted objects, wherein the records identify at least object recipes for the impacted similarity groups and subgroups and wherein the impacted similarity groups and the impacted sub-groups are identified from slice recipes associated with the object recipes.
17. The non-transitory storage medium of claim 12 , further comprising:
identifying a highest numbered sub-group of a particular impacted similarity group, the highest numbered sub-group being marked for cleaning; and
adding a new sub-group to the particular similarity group, wherein the normal operation is directed to the new sub-group and no incoming write operations are directed to the impacted sub-groups that are locked.
18. The non-transitory storage medium of claim 12 , further comprising:
deduplicating one or more objects written to an object storage with respect to sub-groups that are not write locked in the impacted similarity groups, the one or more objects being written to an object storage during the garbage collection operation.