Rule-based re-tiering of incremental backup data stored on a cloud-based object storage
Described is a system for rule-based re-tiering of backup data stored on a cloud-based object storage. More specifically, the system may re-tier objects based on one or more storage rules such that the system retains the ability to efficiently perform a full restore of backup data even when incremental backups are performed to a cloud-based object storage. To provide such a capability, the system may maintain a specialized metadata database that stores information indicating the backup time for each backup, and a list of objects required to perform a full restore to each of the backup times. Accordingly, when initiating a re-tiering based on one or more storage rules, the system may intelligently select candidate objects for re-tiering by leveraging the metadata database to ensure that objects that may still need to be referenced are not unnecessarily moved to a lower storage tier.
1. A system comprising:
one or more processors; and
a non-transitory computer-readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
perform, by a server, a set of backups of client data stored on a client device to a cloud-based object storage, the set of backups of the client data stored as objects within a first storage tier of the object storage;
detect that an amount of storage used by the objects within the first storage tier exceeds a storage limit;
determine a number of the objects to re-tier to a second storage tier of the object storage based on the amount of storage that exceeds the storage limit;
select a threshold time based on the number of the objects to re-tier;
identify, amongst the objects stored by the object storage, a first set of objects not referenced by a first set of backups, amongst the set of backups, performed on or after the threshold time; and
initiate a re-tiering by moving the first set of objects from the first storage tier to the second storage tier of the object storage.
2. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
store, as part of a metadata database, a point-in-time for each backup of the set of backups, and a list of objects referenced by each backup of the set of backups, the list of objects referenced specifying the objects required to perform a full restore to the point-in-time of the corresponding backup.
3. The system of claim 1 , wherein the selecting the threshold time includes determining an approximate amount of storage used by the first set of objects corresponds to the amount of storage that exceeds the storage limit.
4. The system of claim 1 , wherein the detecting that the amount of storage used by the objects within the first storage tier exceeds the storage limit includes:
detecting the amount of storage used by the objects within the first storage tier exceeds, as the storage limit, a predetermined storage allocation for an amount of the client data stored within the first storage tier relative to an amount of the client data stored within the second storage tier.
5. The system of claim 1 , wherein the detecting that the amount of storage used by the objects within the first storage tier exceeds the storage limit includes:
detecting a storage cost associated with the amount of storage used by the objects within the first storage tier; and
recognizing the determined storage cost exceeds, as the storage limit, a predetermined storage cost limit.
6. The system of claim 1 , wherein the identifying the first set of objects includes:
identifying, amongst the set of backups, the first set of backups performed on or after the threshold time, and a second set of backups performed before the threshold time;
identifying, amongst the objects stored by the object storage, a second set of objects referenced by the first set of backups, and a third set of objects referenced by the second set of backups; and
identifying, amongst the third set of objects, the first set of objects as the objects not included in the second set of objects.
7. The system of claim 1 , wherein the selecting the threshold time further includes:
determining, amongst the set of backups, a most recent backup; and
selecting a time that corresponds to the most recent backup as the threshold time.
8. The system of claim 1 , wherein the set of backups include full and incremental backups of the client data.
9. A method comprising:
performing, by a server, a set of backups of client data stored on a client device to a cloud-based object storage, the set of backups of the client data stored as objects within a first storage tier of the object storage;
detecting that an amount of storage used by the objects within the first storage tier exceeds a storage limit;
determining a number of the objects to re-tier to a second storage tier of the object storage based on the amount of storage that exceeds the storage limit;
selecting a threshold time based on the number of the objects to re-tier;
identifying, amongst the objects stored by the object storage, a first set of objects not referenced by a first set of backups, amongst the set of backups, performed on or after the threshold time; and
initiating a re-tiering by moving the first set of objects from the first storage tier to the second storage tier of the object storage.
10. The method of claim 9 , further comprising:
storing, as part of a metadata database, a point-in-time for each backup of the set of backups, and a list of objects referenced by each backup of the set of backups, the list of objects referenced specifying the objects required to perform a full restore to the point-in-time of the corresponding backup.
11. The method of claim 9 , wherein the selecting the threshold time includes determining an approximate amount of storage used by the first set of objects corresponds to the amount of storage that exceeds the storage limit.
12. The method of claim 9 , wherein the detecting that the amount of storage used by the objects within the first storage tier exceeds the storage limit includes:
detecting the amount of storage used by the objects within the first storage tier exceeds, as the storage limit, a predetermined storage allocation for an amount of the client data stored within the first storage tier relative to an amount of the client data stored within the second storage tier.
13. The method of claim 9 , wherein the detecting that the amount of storage used by the objects within the first storage tier exceeds the storage limit includes:
detecting a storage cost associated with the amount of storage used by the objects within the first storage tier; and
recognizing the determined storage cost exceeds, as the storage limit, a predetermined storage cost limit.
14. The method of claim 9 , wherein the identifying the first set of objects includes:
identifying, amongst the set of backups, the first set of backups performed on or after the threshold time, and a second set of backups performed before the threshold time;
identifying, amongst the objects stored by the object storage, a second set of objects referenced by the first set of backups, and a third set of objects referenced by the second set of backups; and
identifying, amongst the third set of objects, the first set of objects as the objects not included in the second set of objects.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
perform, by a server, a set of backups of client data stored on a client device to a cloud-based object storage, the set of backups of the client data stored as objects within a first storage tier of the object storage;
detect that an amount of storage used by the objects within the first storage tier exceeds a storage limit;
determine a number of the objects to re-tier to a second storage tier of the object storage based on the amount of storage that exceeds the storage limit;
select a threshold time based on the number of the objects to re-tier;
identify, amongst the objects stored by the object storage, a first set of objects not referenced by a first set of backups, amongst the set of backups, performed on or after the threshold time; and
initiate a re-tiering by moving the first set of objects from the first storage tier to the second storage tier of the object storage.
16. The computer program product of claim 15 , wherein the program code includes further instructions to:
store, as part of a metadata database, a point-in-time for each backup of the set of backups, and a list of objects referenced by each backup of the set of backups, the list of objects referenced specifying the objects required to perform a full restore to the point-in-time of the corresponding backup.
17. The computer program product of claim 15 , wherein the selecting the threshold time includes determining an approximate amount of storage used by the first set of objects corresponds to the amount of storage that exceeds the storage limit.
18. The computer program product of claim 15 , wherein the detecting that the amount of storage used by the objects within the first storage tier exceeds the storage limit includes:
detecting the amount of storage used by the objects within the first storage tier exceeds, as the storage limit, a predetermined storage allocation for an amount of the client data stored within the first storage tier relative to an amount of the client data stored within the second storage tier.
19. The computer program product of claim 15 , wherein the detecting that the amount of storage used by the objects within the first storage tier exceeds the storage limit includes:
detecting a storage cost associated with the amount of storage used by the objects within the first storage tier; and
recognizing the determined storage cost exceeds, as the storage limit, a predetermined storage cost limit.
20. The computer program product of claim 15 , wherein the identifying the first set of objects includes:
identifying, amongst the set of backups, the first set of backups performed on or after the threshold time, and a second set of backups performed before the threshold time;
identifying, amongst the objects stored by the object storage, a second set of objects referenced by the first set of backups, and a third set of objects referenced by the second set of backups; and
identifying, amongst the third set of objects, the first set of objects as the objects not included in the second set of objects.