Cold tiering microservice for deduplicated data
One example method includes identifying objects that each include one or more segments to be transferred from a source storage tier to a target storage tier, determining a total amount of data to be transferred, using a tiering controller to create worker nodes operable to transfer the segments to the target storage tier, where a number of worker nodes created is based on the amount of data, transferring, from the source storage tier to the target storage tier, only those segments of the objects not already present in the target storage tier, and the transferring of the segments is performed by the worker nodes, and for each of the objects, placing metadata associated with that object in a bucket.
1. A method, comprising:
identifying objects that each include one or more segments to be transferred from a source storage tier to a target storage tier, wherein the source storage tier comprises an active bucket for storing the one or more segments, the source storage tier further comprises a cold bucket, and the target storage tier comprises an active tier and a cold tier, which provides slower access to objects stored in the cold tier than the active tier provides to objects stored in the active tier;
determining, based on the objects identified, an amount of data to be transferred;
using a tiering controller to create worker nodes operable to transfer the segments to the target storage tier;
partitioning the objects across the workers, wherein the objects are partitioned across the workers to avoid repeated readout of the objects from storage;
transferring, from the source storage tier to the target storage tier, only those segments of the objects not already present in the target storage tier as a compression region in the target storage tier, and the transferring of the segments is performed by the worker nodes; and
placing metadata associated with objects, which are stored in the cold tier, in the cold bucket of the source storage tier.
2. The method as recited in claim 1 , wherein the tiering controller and the workers comprise respective microservices operating as part of a scalable container orchestration system.
3. The method as recited in claim 1 , wherein the workers identify the segments that are to be transferred.
4. The method as recited in claim 3 , wherein the segments are identified based on a comparison of fingerprints of a similarity group with fingerprints included in a slice recipe.
5. The method as recited in claim 1 , wherein identification of the objects is based on receipt of user input designating those objects.
6. The method as recited in claim 1 , further comprising performing a garbage collection process that periodically releases any unreferenced structures in the active tier.
7. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
identifying objects that each include one or more segments to be transferred from a source storage tier to a target storage tier, wherein the source storage tier comprises an active bucket for storing the one or more segments, the source storage tier further comprises a cold bucket, and the target storage tier comprises an active tier and a cold tier, which provides slower access to objects stored in the cold tier than the active tier provides to objects stored in the active tier;
determining, based on the objects identified, an amount of data to be transferred;
using a tiering controller to create worker nodes operable to transfer the segments to the target storage tier;
partitioning the objects across the workers, wherein the objects are partitioned across the workers to avoid repeated readout of the objects from storage;
transferring, from the source storage tier to the target storage tier, only those segments of the objects not already present in the target storage tier as a compression region in the target storage tier, and the transferring of the segments is performed by the worker nodes; and
placing metadata associated with objects, which are stored in the cold tier, in the cold bucket of the source storage tier.
8. The non-transitory storage medium as recited in claim 7 , wherein the tiering controller and the workers comprise respective microservices operating as part of a scalable container orchestration system.
9. The non-transitory storage medium as recited in claim 7 , wherein the workers identify the segments that are to be transferred.
10. The non-transitory storage medium as recited in claim 9 , wherein the segments are identified based on a comparison of fingerprints of a similarity group with fingerprints included in a slice recipe.
11. The non-transitory storage medium as recited in claim 7 , wherein identification of the objects is based on receipt of user input designating those objects.
12. The non-transitory storage medium as recited in claim 7 , wherein the operations further comprise performing a garbage collection process that periodically releases any unreferenced structures in the active tier.