Efficient object storage management when performing backups to a cloud-based object storage
Described is a system (and method) for efficient object storage management when backing up data to a cloud-based object storage. The system may be implemented as part of a server (or gateway) that provides a backup service to a client device by acting as an intermediary when backing up data from the client device to a third-party cloud-based object storage. The system may implement various specialized procedures to efficiently store backup data as objects within the object storage. The procedures may include packing client data into objects of a consistent size to improve storage performance. The system may also improve storage performance and conserve storage by analyzing the data stored within an object and reallocating the data as necessary. More particularly, the system may efficiently reallocate data to new objects when the amount of live data within an object falls below a predetermined threshold.
1. A system comprising:
one or more processors; and
a non-transitory computer-readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
perform a first backup of client data stored on a client device at a first point-in-time to a cloud-based object storage, the first backup including
storing the client data within a first object on the object storage, and
storing a first set of metadata identifying the client data stored within the first object as part of a metadata database;
perform a second backup of the client data stored on the client device at a second point-in-time to the object storage, the second backup including
storing new client data since the first point-in-time within a second object on the object storage,
storing a second set of metadata identifying the new client data stored within the second object as part of the metadata database, and
storing a third set of metadata identifying the client data that has been deleted from the client device since the first backup as part of the metadata database;
determine whether an amount of live data remaining within the first object falls below a predetermined threshold;
create a third object within the object storage to store the remaining live data, in response to determining the amount of live data remaining within the first object falls below the predetermined threshold; and
update the metadata database to identify the remaining live data is stored within the third object.
2. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine whether the client data stored in the first object expires before the client data stored in the second object, and
wherein the third object is created in further response to determining the client data stored in the first object expires before the client data stored in the second object.
3. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
delete the first object in response to creating the third object to store the remaining live data.
4. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
identify the client data that has been deleted since the first backup is stored within the first object based on a search of the metadata database.
5. The system of claim 1 , wherein the determining whether the amount of live data remaining within the first object falls below the predetermined threshold is in response to recognizing that client data has been deleted from the client device since the first backup.
6. The system of claim 1 , wherein the determining whether the amount of live data remaining within the first object falls below the predetermined threshold includes:
retrieving, from the metadata database, at least some of the first set of metadata identifying the client data stored within the first object and the third set of metadata identifying the client data that has been deleted since the first backup; and
determining the amount of live data remaining within the first object based on data size information obtained from the retrieved first set of metadata and the retrieved third set of metadata.
7. The system of claim 6 , wherein the predetermined threshold is based on the amount of live data remaining within the first object compared to an amount of the client data stored within the first object that has been deleted since the first backup.
8. A method comprising:
performing a first backup of client data stored on a client device at a first point-in-time to a cloud-based object storage, the first backup including
storing the client data within a first object on the object storage, and
storing a first set of metadata identifying the client data stored within the first object as part of a metadata database;
performing a second backup of the client data stored on the client device at a second point-in-time to the object storage, the second backup including
storing new client data since the first point-in-time within a second object on the object storage,
storing a second set of metadata identifying the new client data stored within the second object as part of the metadata database, and
storing a third set of metadata identifying the client data that has been deleted from the client device since the first backup as part of the metadata database;
determining whether an amount of live data remaining within the first object falls below a predetermined threshold;
creating a third object within the object storage to store the remaining live data, in response to determining the amount of live data remaining within the first object falls below the predetermined threshold; and
updating the metadata database to identify the remaining live data is stored within the third object.
9. The method of claim 8 , further comprising:
determining whether the client data stored in the first object expires before the client data stored in the second object, and
wherein the third object is created in further response to determining the client data stored in the first object expires before the client data stored in the second object.
10. The method of claim 8 , further comprising:
deleting the first object in response to creating the third object to store the remaining live data.
11. The method of claim 8 , further comprising:
identifying the client data that has been deleted since the first backup is stored within the first object based on a search of the metadata database.
12. The method of claim 8 , wherein the determining whether the amount of live data remaining within the first object falls below the predetermined threshold is in response to recognizing that client data has been deleted from the client device since the first backup.
13. The method of claim 8 , wherein the determining whether the amount of live data remaining within the first object falls below the predetermined threshold includes:
retrieving, from the metadata database, at least some of the first set of metadata identifying the client data stored within the first object and the third set of metadata identifying the client data that has been deleted since the first backup; and
determining the amount of live data remaining within the first object based on data size information obtained from the retrieved first set of metadata and the retrieved third set of metadata.
14. The method of claim 13 , wherein the predetermined threshold is based on the amount of live data remaining within the first object compared to an amount of the client data stored within the first object that has been deleted since the first backup.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
perform a first backup of client data stored on a client device at a first point-in-time to a cloud-based object storage, the first backup including
storing the client data within a first object on the object storage, and
storing a first set of metadata identifying the client data stored within the first object as part of a metadata database;
perform a second backup of the client data stored on the client device at a second point-in-time to the object storage, the second backup including
storing new client data since the first point-in-time within a second object on the object storage,
storing a second set of metadata identifying the new client data stored within the second object as part of the metadata database, and
storing a third set of metadata identifying the client data that has been deleted from the client device since the first backup as part of the metadata database;
determine whether an amount of live data remaining within the first object falls below a predetermined threshold;
create a third object within the object storage to store the remaining live data, in response to determining the amount of live data remaining within the first object falls below the predetermined threshold; and
update the metadata database to identify the remaining live data is stored within the third object.
16. The computer program product of claim 15 , wherein the program code includes further instructions to:
determine whether the client data stored in the first object expires before the client data stored in the second object, and
wherein the third object is created in further response to determining the client data stored in the first object expires before the client data stored in the second object.
17. The computer program product of claim 15 , wherein the program code includes further instructions to:
delete the first object in response to creating the third object to store the remaining live data.
18. The computer program product of claim 15 , wherein the program code includes further instructions to:
identify the client data that has been deleted since the first backup is stored within the first object based on a search of the metadata database.
19. The computer program product of claim 15 , wherein the determining whether the amount of live data remaining within the first object falls below the predetermined threshold is in response to recognizing that client data has been deleted from the client device since the first backup.
20. The computer program product of claim 15 , wherein the determining whether the amount of live data remaining within the first object falls below the predetermined threshold includes:
retrieving, from the metadata database, at least some of the first set of metadata identifying the client data stored within the first object and the third set of metadata identifying the client data that has been deleted since the first backup; and
determining the amount of live data remaining within the first object based on data size information obtained from the retrieved first set of metadata and the retrieved third set of metadata, and
wherein the predetermined threshold is based on the amount of live data remaining within the first object compared to an amount of the client data stored within the first object that has been deleted since the first backup.