Marking impacted similarity groups in garbage collection operations in deduplicated storage systems
Systems and methods for marking similarity groups impacted by a garbage collection operation are disclosed. Similarity groups are used to identify segments associated with objects in a computing system. Using deletion records that identify objects to be deleted, the similarity groups impacted by the deletion records can be identified. The live segments associated with the impacted similarity groups are also identified. This allows segments that are associated with the deleted objects and that are not associated with any live objects to be removed.
1. A method for marking similarity groups impacted by a garbage collection operation performed in a storage system, the method comprising:
receiving input identifying objects to be deleted, wherein each of the objects is associated with a recipe;
removing recipes of the identified objects from an object storage of the storage system to a deletion storage of the storage system such that the identified objects are not visible to users, wherein each of the removed recipes corresponds to a deletion record;
receiving a list of the deletion records that identify the objects that are to be deleted from the storage system;
identifying slice recipes for the objects to be deleted from the deletion records;
marking similarity groups associated with the slice recipes as impacted similarity groups that are associated with segments of the objects in the deletion records;
generating a mapping of the impacted similarity groups to a map structure that identifies live segments in the impacted similarity groups, the mapping defining an overall size of memory needed for the impacted similarity groups, wherein the map structure allows dead segments associated with the objects to be deleted to be identified and removed from the impacted similarity groups; and
estimating a number of a plurality of workers to be used to delete the identified dead segments from the storage system using a number of input/output operations used to clean a single impacted similarity group of the impacted similarity groups.
2. The method of claim 1 , further comprising cleaning the impacted similarity groups.
3. The method of claim 2 , further comprising cleaning compression regions associated with the impacted similarity groups or cleaning only impacted compression regions associated with the impacted similarity groups.
4. The method of claim 1 , wherein the list of deletion records includes objects that were fully written to the storage system.
5. The method of claim 1 , further comprising identifying the live segments associated with the impacted similarity groups by evaluating the live objects.
6. The method of claim 5 , generating the map structure that maps the live segments to compression regions of the impacted similarity groups, wherein segments in the map structure that are not marked correspond to dead segments in the compression regions of the impacted similarity groups.
7. The method of claim 5 , further comprising allocating the live objects to the plurality of workers for evaluation and allocating the impacted similarity groups to the plurality of workers configured to clean the impacted similarity groups.
8. The method of claim 7 , further comprising cleaning the impacted similarity groups on a basis of impacted compression regions, wherein live segments in the impacted compression regions are carried forward to a new compression region and the impacted compression regions that have been carried forward are deleted in their entirety from the storage system.
9. The method of claim 1 , further comprising removing the deletion records.
10. A non-transitory computer readable medium comprising computer executable instructions for implementing a method for marking similarity groups impacted by a garbage collection operation performed in a storage system, the method comprising:
receiving input identifying objects to be deleted, wherein each of the objects is associated with a recipe;
removing recipes of the identified objects from an object storage of the storage system to a deletion storage of the storage system such that the identified objects are not visible to users, wherein each of the removed recipes corresponds to a deletion record;
receiving a list of the deletion records that identify the objects that are to be deleted from the storage system;
identifying slice recipes for the objects to be deleted from the deletion records;
marking similarity groups associated with the slice recipes as impacted similarity groups that are associated with segments of the objects in the deletion records;
generating a mapping of the impacted similarity groups to a map structure that identifies live segments in the impacted similarity groups, the mapping defining an overall size of memory needed for the impacted similarity groups, wherein the map structure allows dead segments associated with the objects to be deleted to be identified and removed from the impacted similarity groups; and
estimating a number of a plurality of workers to be used to delete the identified dead segments from the storage system using a number of input/output operations used to clean a single impacted similarity group of the impacted similarity groups.
11. The method of claim 10 , further comprising cleaning the impacted similarity groups.
12. The method of claim 11 , further comprising cleaning compression regions associated with the impacted similarity groups or cleaning only impacted compression regions associated with the impacted similarity groups.
13. The method of claim 10 , wherein the list of deletion records includes objects that were not fully written to the storage system.
14. The method of claim 10 , further comprising identifying live segments associated with the impacted similarity groups by evaluating the live objects.
15. The method of claim 14 , generating the map structure that maps the live segments to compression regions of the impacted similarity groups, wherein segments in the map structure that are not marked correspond to dead segments in the compression regions of the impacted similarity groups.
16. The method of claim 15 , further comprising allocating the live objects to the plurality of workers for evaluation and allocating the impacted similarity groups to the plurality of workers configured to clean the impacted similarity groups.
17. The method of claim 16 , further comprising cleaning the impacted similarity groups on a basis of impacted compression regions, wherein live segments in the impacted compression regions are carried forward to a new compression region and the impacted compression regions that have been carried forward are deleted in their entirety from the storage system.
18. The method of claim 10 , further comprising removing the deletion records.
19. A method for marking similarity groups impacted by a garbage collection operation performed in a storage system, the method comprising:
receiving a list of deletion records that identify objects that are to be deleted from the storage system;
estimating a number of a plurality of workers to be used to delete the identified objects from the storage system;
instantiating the number of the plurality of workers;
identifying similarity groups from the deletion records as impacted similarity groups, wherein the impacted similarity groups store segments associated with the objects that are to be deleted in compression regions;
evaluating live objects or live recipes of the live objects to identify impacted live objects that are associated with the impacted similarity groups;
determining a size of the impacted similarity groups;
estimating a number of a plurality of workers to be used to delete the segments associated with the objects that are to be deleted using a number of input/output operations used to clean a single impacted similarity group of the impacted similarity groups;
instantiating the number of the plurality of workers;
assigning a range of the similarity groups to each of the workers based upon the size of the impacted similarity groups;
generating map structures for the compression regions in the impacted similarity group;
identifying live segments from the live objects or the live recipes and marking the map structures accordingly such that the map structures identify which of the segments in the compression regions are live segments and which of the segments are dead segments, wherein the dead segments are not referenced by any of the live objects or the recipes of the live objects;
cleaning the compression regions of the dead segments by the plurality of workers such that only the live segments remain.
20. The method of claim 19 , further comprising cleaning only those compression regions where a ratio of dead segments to live segments is greater than a predetermined threshold and wherein evaluating live objects or live recipes comprises evaluating the live objects or live recipes by the number of the plurality of workers.