Estimating worker nodes needed for performing garbage collection operations
Systems and methods for estimating the number of workers needed to perform a garbage collection operation are disclosed. Similarity groups are used to identify segments associated with objects in a computing system. Using deletion records that identify objects to be deleted, the similarity groups impacted by the deletion records can be identified. The number of workers can be determined based on the impacted similarity groups. More specifically, the number of impacted similarity groups and/or workers can be evaluated in terms of memory requirements, input/output constraints and/or time requirements to estimate the number or workers needed to clean similarity groups impacted by a garbage collection operation.
1. A method for performing a garbage collection operation, the method comprising:
instantiating a controller that estimates a number of workers needed for the garbage collection operation; and
instantiating and allocating workers, wherein the workers;
identify impacted similarity groups within a compression region, wherein the impacted similarity groups include one or more segments that should be deleted;
mark the impacted similarity groups;
write-lock the impacted similarity groups;
mark live segments within an impacted similarity group; and
copy forward the live segments into a new compression region.
2. The method of claim 1 , wherein the controller is created within a Kubernetes cluster.
3. The method of claim 1 , wherein the impacted similarity groups are identified from a deletion bucket, which stores information related to deleted objects.
4. The method of claim 1 , wherein the information related to deleted objects includes slices associated with deleted objects and the impacted similarity groups can be determined from the slices associated with deleted objects.
5. The method of claim 1 , wherein the controller allocates assignments to the workers.
6. The method of claim 1 , wherein the controller estimates the number of workers using factors including one or more of an environmental variable, worker memory, impacted similarity groups, or IO operations.
7. The method of claim 1 , wherein the controller;
accesses deletion records to retrieve or access slice recipes;
identifies impacted similarity groups and determine the sizes of the impacted similarity groups;
estimates the number of workers needed to clean the impacted similarity groups; and
prepares worker assignments based on a number of the impacted similarity groups and a size of the impacted similarity groups.
8. The method of claim 1 , wherein while identifying the impacted similarity groups, the impacted similarity groups are mapped to data structures that include the live segments.
9. The method of claim 1 , wherein the impacted similarity groups include one or more subgroups.
10. The method of claim 1 , wherein the worker marks live fingerprints segments by obtaining slice recipes for slices of live objects, identifying from the slice recipe whether the similarity group associated with that slice is an impacted similarity group, and recording the live fingerprint segment in a live segment structure.
11. The method of claim 10 , wherein the live segment structure contains information including a number of live slices, the number of live segments, and a list of which segments are live.
12. The method of claim 1 , wherein workers only copy forward the live segments into the new compression region when a ratio of the live segments to segments to be deleted falls below a predetermined threshold.
13. The method of claim 1 , wherein the workers delete the compression region after the live segments have been copied forward to the new compression region.
14. A non-transitory computer readable medium comprising computer executable instructions that, when executed, perform a method of a garbage collection operation, the method comprising:
instantiating a controller that estimates a number of workers needed for the garbage collection operation, wherein the controller;
accesses deletion records to retrieve or access slice recipes;
identifies impacted similarity groups from the deletion records and determines a size of the impacted similarity groups;
estimates the number of workers needed to clean the impacted similarity groups; and
prepares worker assignments based on a number of the impacted similarity groups and the size of the impacted similarity groups; and
instantiating and allocating workers, wherein the workers;
identify compression regions that include the impacted similarity groups, wherein the impacted similarity groups includes one or more segments to be deleted; and
for each of the identified compression regions:
mark the impacted similarity groups;
write-lock the impacted similarity groups;
mark live segments within the impacted similarity groups; and
copy forward the live segments into a new compression region.
15. The non-transitory computer readable medium of claim 14 , wherein the impacted similarity groups are identified from a deletion bucket, which stores information related to deleted objects.
16. The non-transitory computer readable medium of claim 14 , wherein the information related to deleted objects includes slices associated with deleted objects and the impacted similarity groups can be determined from the slices associated with deleted objects.
17. The non-transitory computer readable medium of claim 14 , wherein the controller estimates the number of workers using factors including one or more of an environmental variable, worker memory, impacted similarity groups, or IO operations.
18. The non-transitory computer readable medium of claim 14 , wherein while identifying impacted similarity groups, the impacted similarity groups are mapped to data structures that include live segments.
19. The non-transitory computer readable medium of claim 14 , wherein the worker marks live fingerprints segments by obtaining slice recipes for slices of live objects, identifying from the slice recipe whether the similarity group associated with that slice is an impacted similarity group, and records the live fingerprint segment in a live segment structure.
20. The non-transitory computer readable medium of claim 14 , wherein workers only copy forward the live segments into the new compression region when a ratio of the live segments to segments to be deleted falls below a predetermined threshold.