Leveraging metadata to manage the expiration of objects storing incremental backup data
Described is a system for managing the expiration of objects storing incremental backup data by leveraging specialized metadata. The system may provide a layer of intelligence when freeing up storage space on an object storage to ensure that a full recovery of client backup data may be performed when storing incremental backup data. The system may maintain metadata indicating the latest and/or earliest backups that reference particular objects. The system may then leverage this information to mark particular objects for removal (e.g. deletion or re-tiering). For example, the system may access the metadata to identify which objects stored by a previous (e.g. expired) backup are still referenced by a subsequent (e.g. non-expired) backup. As a result, the system may ensure that objects are not removed prematurely and that a full recovery of the client data to a point-in-time of any incremental backup may be performed.
1. A system comprising:
one or more processors; and
a non-transitory computer-readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
perform a first backup of client data stored on a client device at a first point-in-time to an object storage, the client data at the first point-in-time including a first file and a second file, the first backup including creating a first object within the object storage to store a backup of the first file and a second object within the object storage to store a backup of the second file;
update a metadata database, in response to the first backup, to indicate the first backup as a latest backup to reference the first object, and the second object;
perform a second backup of the client data stored on the client device at a second point-in-time to the object storage, the client data at the second point-in-time including a new third file, the second backup including creating a third object within the object storage to store a backup of the third file;
update the metadata database, in response to the second backup, to indicate the second backup as the latest backup to reference the second object, and the third object;
obtain an expiration time associated with the performed backups, the expiration time used to free up storage space on the object storage;
identify the first backup as expired based on a determination the first point-in-time precedes the expiration time; and
identify, amongst the objects created during the first backup, the first object as expired based on a determination the latest backup to reference the first object is the expired first backup.
2. The system of claim 1 , wherein the client data at the second point-in-time further includes a deletion of the first file from the client device since the first point-in-time.
3. The system of claim 2 , wherein updating the metadata database, in response to the second backup, does not include indicating the second backup as the latest backup to reference the first object in response to the deletion of the first file.
4. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
update the metadata database, in response to the first backup, to indicate the first backup as an earliest backup to reference the first object, and the second object; and
update the metadata database, in response to the second backup, to indicate the second backup as the earliest backup to reference the third object.
5. The system of claim 4 , wherein identifying the first object as expired is further based on a determination the first backup is both the earliest backup to reference the first object and the latest backup to reference the first object.
6. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
mark the first object for re-tiering or deletion, in response to identifying the first object as expired.
7. The system of claim 1 , wherein the second backup is an incremental backup.
8. A method comprising:
performing a first backup of client data stored on a client device at a first point-in-time to an object storage, the client data at the first point-in-time including a first file and a second file, the first backup including creating a first object within the object storage to store a backup of the first file and a second object within the object storage to store a backup of the second file;
updating a metadata database, in response to the first backup, to indicate the first backup as a latest backup to reference the first object, and the second object;
performing a second backup of the client data stored on the client device at a second point-in-time to the object storage, the client data at the second point-in-time including a new third file, the second backup including creating a third object within the object storage to store a backup of the third file;
updating the metadata database, in response to the second backup, to indicate the second backup as the latest backup to reference the second object, and the third object;
obtaining an expiration time associated with the performed backups, the expiration time used to free up storage space on the object storage;
identifying the first backup as expired based on a determination the first point-in-time precedes the expiration time; and
identifying, amongst the objects created during the first backup, the first object as expired based on a determination the latest backup to reference the first object is the expired first backup.
9. The method of claim 8 , wherein the client data at the second point-in-time further includes a deletion of the first file from the client device since the first point-in-time.
10. The method of claim 9 , wherein updating the metadata database, in response to the second backup, does not include indicating the second backup as the latest backup to reference the first object in response to the deletion of the first file.
11. The method of claim 8 , further comprising:
updating the metadata database, in response to the first backup, to indicate the first backup as an earliest backup to reference the first object, and the second object; and
updating the metadata database, in response to the second backup, to indicate the second backup as the earliest backup to reference the third object.
12. The method of claim 11 , wherein identifying the first object as expired is further based on a determination the first backup is both the earliest backup to reference the first object and the latest backup to reference the first object.
13. The method of claim 8 , further comprising:
marking the first object for re-tiering or deletion, in response to identifying the first object as expired.
14. The method of claim 8 , wherein the second backup is an incremental backup.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
perform a first backup of client data stored on a client device at a first point-in-time to an object storage, the client data at the first point-in-time including a first file and a second file, the first backup including creating a first object within the object storage to store a backup of the first file and a second object within the object storage to store a backup of the second file;
update a metadata database, in response to the first backup, to indicate the first backup as a latest backup to reference the first object, and the second object;
perform a second backup of the client data stored on the client device at a second point-in-time to the object storage, the client data at the second point-in-time including a new third file, the second backup including creating a third object within the object storage to store a backup of the third file;
update the metadata database, in response to the second backup, to indicate the second backup as the latest backup to reference the second object, and the third object;
obtain an expiration time associated with the performed backups, the expiration time used to free up storage space on the object storage;
identify the first backup as expired based on a determination the first point-in-time precedes the expiration time; and
identify, amongst the objects created during the first backup, the first object as expired based on a determination the latest backup to reference the first object is the expired first backup.
16. The computer program product of claim 15 , wherein the client data at the second point-in-time further includes a deletion of the first file from the client device since the first point-in-time.
17. The computer program product of claim 16 , wherein updating the metadata database, in response to the second backup, does not include indicating the second backup as the latest backup to reference the first object in response to the deletion of the first file.
18. The computer program product of claim 15 , wherein the program code includes further instructions to:
update the metadata database, in response to the first backup, to indicate the first backup as an earliest backup to reference the first object, and the second object; and
update the metadata database, in response to the second backup, to indicate the second backup as the earliest backup to reference the third object.
19. The computer program product of claim 18 , wherein identifying the first object as expired is further based on a determination the first backup is both the earliest backup to reference the first object and the latest backup to reference the first object.
20. The computer program product of claim 15 , wherein the program code includes further instructions to:
mark the first object for re-tiering or deletion, in response to identifying the first object as expired.