Flexible tiering of snapshots to archival storage in remote object stores
Techniques are provided for tiering snapshots to archival storage in remote object stores. A restore time metric, indicating that objects comprising snapshot data of snapshots created within a threshold timespan are to be available within a storage tier of a remote object store for performing restore operations, may be identified. A scanner may be executed to evaluate snapshots using the restore time metric to identify a set of candidate snapshots for archival from the storage tier to an archival storage tier of the remote object store. For each candidate snapshot within the set of candidate snapshots, the scanner may evaluate metadata associated with the candidate snapshot to identity one or more objects eligible for archival from the storage tier to the archival storage tier, and may archive the one or more objects from the storage tier to the archival storage tier.
1. A method implemented by a scanner to perform operations comprising:
identifying a candidate snapshot for archival from a storage tier to an archival storage tier of a remote object store based upon the candidate snapshot being older than a restore time metric;
executing the scanner to evaluate an object identifier map of the candidate snapshot to identify objects within which snapshot data of the candidate snapshot are stored, wherein the object identifier map comprises bits set to indicate which objects within the remote object store comprise the snapshot data of the candidate snapshot;
evaluating, by the scanner, the object identifier map of the candidate snapshot and object identifier maps of non-candidate snapshots that are not older than the restore time metric to identify overlapping objects that store the snapshot data of the candidate snapshot and snapshot data of at least one non-candidate snapshot and to identify non-overlapping objects that store the snapshot data of the candidate snapshot and do not store snapshot data of any non-candidate snapshots; and
archiving the non-overlapping objects from the storage tier to the archival storage tier.
2. The method of claim 1 , wherein the evaluating comprises:
performing a difference operation, by the scanner, upon the object identifier map of the candidate snapshot and the object identifier maps of the non-candidate snapshots.
3. The method of claim 1 , comprising:
identifying an object as a non-overlapping object for archival based upon the object storing snapshot data of candidate snapshots older than the restore time metric and not storing snapshot data of non-candidate snapshots.
4. The method of claim 1 , comprising:
identifying an object as a non-overlapping object for archival based upon the object not storing snapshot data of a non-candidate snapshot that is not older than the restore time metric.
5. The method of claim 1 , wherein the non-overlapping objects do not comprise snapshot data of the non-candidate snapshots that are not older than the restore time metric.
6. The method of claim 1 , comprising:
identifying an overlapping object based upon the overlapping object storing snapshot data of a non-candidate snapshot that is not older than the restore time metric.
7. The method of claim 1 , wherein the operations comprise:
performing, by the scanner, a difference upon bits within the object identifier map of the candidate snapshot and bits within the object identifier maps of the non-candidate snapshots.
8. A computing device comprising:
a memory comprising instructions for implementing a scanner; and
a processor coupled to the memory, the processor configured to execute the instructions to cause the processor to perform operations comprising:
identifying a candidate snapshot for archival from a storage tier to an archival storage tier of a remote object store based upon the candidate snapshot being older than a restore time metric;
executing the scanner to evaluate an object identifier map of the candidate snapshot to identify objects within which snapshot data of the candidate snapshot are stored, wherein the object identifier map comprises bits set to indicate which objects within the remote object store comprise the snapshot data of the candidate snapshot;
evaluating, by the scanner, the object identifier map of the candidate snapshot and object identifier maps of non-candidate snapshots that are not older than the restore time metric to identify overlapping objects that store the snapshot data of the candidate snapshot and snapshot data of at least one non-candidate snapshot and to identify non-overlapping objects that store the snapshot data of the candidate snapshot and do not store snapshot data of any non-candidate snapshots selecting; and
archiving the non-overlapping objects from the storage tier to the archival storage tier.
9. The computing device of claim 8 , wherein the operations comprise:
performing a difference operation, by the scanner, upon the object identifier map of the candidate snapshot and the object identifier maps of the non-candidate snapshots.
10. The computing device of claim 8 , wherein the operations comprise:
evaluating, by the scanner, a first object identifier map of the snapshot, a second object identifier map of a prior snapshot, and a third object identifier map of a next snapshot to identify the non-overlapping objects for archival.
11. The computing device of claim 10 , wherein the scanner identifies differences between the first object identifier map, the second object identifier map, and the third object identifier map to identify the non-overlapping objects eligible for archival.
12. The computing device of claim 8 , wherein the operations comprise:
executing, difference operations by the scanner according to a first referenced mode of operation, to identify objects that are first referenced by a snapshot as the non-overlapping objects eligible for archival, wherein the scanner evaluates a set of snapshots to identify the snapshot.
13. The computing device of claim 8 , wherein the operations comprise:
executing, difference operations by the scanner according to a last referenced mode of operation, to identify objects that are last referenced by a snapshot as the non-overlapping objects eligible for archival, wherein the scanner evaluates a set of snapshots to identify the snapshot.
14. The computing device of claim 8 , wherein the operations comprise:
executing, difference operations by the scanner according to a uniquely referenced mode of operation to identify objects that are uniquely referenced by a snapshot and no other snapshots as the non-overlapping objects eligible for archival, wherein the scanner evaluates a set of snapshots to identify the snapshot.
15. A non-transitory machine readable medium comprising instructions for performing a method, which when executed by a machine, causes the machine to:
identifying a candidate snapshot for archival from a storage tier to an archival storage tier of a remote object store based upon the candidate snapshot being older than a restore time metric;
executing a scanner to evaluate an object identifier map of the candidate snapshot to identify objects within which snapshot data of the candidate snapshot are stored, wherein the object identifier map comprises bits set to indicate which objects within the remote object store comprise the snapshot data of the candidate snapshot;
evaluating, by the scanner, the object identifier map of the candidate snapshot and object identifier maps of non-candidate snapshots that are not older than the restore time metric to identify overlapping objects that store the snapshot data of the candidate snapshot and snapshot data of at least one non-candidate snapshot and to identify non-overlapping objects that store the snapshot data of the candidate snapshot and do not store snapshot data of any non-candidate snapshots; and
archiving the non-overlapping objects from the storage tier to the archival storage tier.
16. The non-transitory machine readable medium of claim 15 , comprising:
performing a difference operation, by the scanner, upon the object identifier map of the candidate snapshot and the object identifier maps of the non-candidate snapshots.
17. The non-transitory machine readable medium of claim 15 , comprising:
identifying an object as a non-overlapping object for archival based upon the object storing snapshot data of candidate snapshots older than the restore time metric and not storing snapshot data of non-candidate snapshots.
18. The non-transitory machine readable medium of claim 15 , comprising:
identifying an object as a non-overlapping object for archival based upon the object not storing snapshot data of a non-candidate snapshot that is not older than the restore time metric.
19. The non-transitory machine readable medium of claim 15 , wherein the non-overlapping objects do not comprise snapshot data of the non-candidate snapshots that are not older than the restore time metric.
20. The non-transitory machine readable medium of claim 15 , comprising:
identifying an overlapping object based upon the overlapping object storing snapshot data of a non-candidate snapshot that is not older than the restore time metric.