Garbage collection for a deduplicated cloud tier using microservices
Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live regions such as compression regions, the dead segments in the dead compression regions are deleted by copying the live compression regions into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a microservices based approach.
1. In a computing system that provides an active tier of data and a cloud tier of data in a cloud, a method for performing a garbage collection operation in the cloud tier, the method comprising:
processing metadata, stored in the active tier, for the cloud tier by a garbage collection engine to identify containers in the cloud tier to be cleaned, wherein the identified containers contain compression regions;
determining from the metadata which of the compression regions in the cloud tier are dead compression regions and which of the compression regions in the cloud tier are live compression regions;
instantiating an instance starter by the garbage collection engine to spin up a microservice in the cloud and spinning up the microservice;
generating recipes that identify locations of the live compression regions in the identified containers, wherein the recipes specify byte ranges of the live compression regions in the identified containers;
sending the recipes to the microservice, wherein the microservice copies data in the byte ranges of the live compression regions to copy to new containers, without performing decryption operations, from the identified containers and then deletes the identified containers; and
spinning down the microservice when the garbage collection operation ends.
2. The method of claim 1 , wherein each recipe identifies at least a container, a location of a live compression region in the container, a size of the live compression region, and a destination container for storing the live compression region.
3. The method of claim 1 , wherein the recipes are distributed to a plurality of microservice instances such that cloud tier of data is cleaned in parallel.
4. The method of claim 1 , further comprising updating the metadata stored in the active tier to reflect the locations of the live compression regions in the new containers stored in the cloud tier after performing the recipes.
5. The method of claim 1 , wherein the metadata includes metadata of L0 and Lp containers stored in the cloud, the metadata of the L0 and Lp containers stored in the cloud including fingerprints of segments in the L0 and Lp containers.
6. The method of claim 5 , further comprising performing a lookup to identify live compression regions and dead compression regions of the Lp containers.
7. The method of claim 6 , further comprising generating the recipes that allow the live compression regions from the Lp containers to be copied into new LP containers.
8. The method of claim 7 , further comprising writing the new Lp containers locally and to the cloud.
9. The method of claim 8 , further comprising copying metadata of the new Lp containers to a new CMETA container, wherein the new CMETA container is written locally and to the cloud.
10. The method of claim 1 , further comprising iterating metadata sections of local CMETA containers to identify the live compression regions of L0 containers.
11. The method of claim 10 , further comprising forming the recipes based on the local CMETA containers.
12. The method of claim 11 , further comprising copying metadata corresponding to the recipes into a new CMETA container locally and replicating the new CMETA container to the cloud.
13. The method of claim 12 , further comprising deleting the Lp, L0 and CMETA containers from which live compression regions were copied forward to reclaim space in the cloud.
14. The method of claim 1 , wherein the live compression regions are copied forward without regard to format, compression status, and encryption status, wherein at least some of the live compression regions include dead segments and wherein only live segments from these live compression regions are copied to the new containers.
15. A non-transitory computer readable medium comprising computer executable instructions that, when executed, perform a garbage collection operation in a cloud tier of data in a cloud associated with a computing system that also has a local tier of data, the method comprising:
processing metadata, stored in the active tier, for the cloud tier by a garbage collection engine to identify containers in the cloud tier to be cleaned, wherein the identified containers contain compression regions;
determining from the metadata which of the compression regions in the cloud tier are dead compression regions and which of the compression regions in the cloud tier are live compression regions;
instantiating an instance starter by the garbage collection engine to spin up a microservice in the cloud and spinning up the microservice;
generating recipes that identify locations of the live compression regions in the identified containers, wherein the recipes specify byte ranges of the live compression regions in the identified containers;
sending the recipes to the microservice, wherein the microservice copies data in the byte ranges of the live compression regions to copy to new containers, without performing decryption operations, from the identified containers and then deletes the identified containers; and
spinning down the microservice when the garbage collection operation ends.
16. The non-transitory computer readable medium of claim 15 , wherein each of the recipes identifies at least an existing container, a location of a live compression region, a size of the live compression region, and a destination container for storing the live compression region.
17. The non-transitory computer readable medium of claim 15 , wherein only live segments in the live compression regions are copied forward to the new containers.
18. The non-transitory computer readable medium of claim 15 , further comprising updating the metadata in the active tier to reflect the cloud tier of data after performing the recipes.
19. The non-transitory computer readable medium of claim 15 , further comprising:
identifying, from the metadata, metadata of L0 and Lp containers stored in the cloud, the metadata of the L0 and Lp containers stored in the cloud including fingerprints of segments in the L0 and Lp containers;
performing a lookup to identify live regions and dead regions of the Lp containers;
generating the recipes that allow the live regions from the Lp containers to be copied into new LP containers;
writing the new Lp containers locally and to the cloud; and
copying metadata of the new Lp containers to a new CMETA container, wherein the new CMETA container is written locally and to the cloud.
20. The method of claim 19 , further comprising:
iterating metadata sections of local CMETA containers to identify live compression regions of L0 containers;
forming the recipes based on the local CMETA containers; and
copying metadata corresponding to the recipes into a new CMETA container locally and replicating the new CMETA container to the cloud.