Specialized data management for performing incremental backups to a cloud-based object storage
Described is a system (and method) for specialized data management when performing incremental backups to a cloud-based object storage. When backing up client data, the server may utilize incremental backups to conserve the amount of storage space required on the object storage. However, the server may implement a configuration that enables full reconstruction of the client data at the time of recovery. To provide such capabilities, the server may leverage a specialized metadata database. The metadata database may be an embedded database maintained by the server. The server may leverage the metadata database to provide further storage efficiencies by storing certain incremental backup data exclusively within the metadata database. Accordingly, the server may implement a data management configuration as part of a backup service that conserves storage usage, and as a result, storage costs when using on a third-party cloud-based object storage.
1. A system comprising:
one or more processors; and
a non-transitory computer-readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
perform, at a first point-in-time, a full backup of client data stored on a client device to a cloud-based object storage, the full backup including:
storing, as a first set of objects associated with the first point-in-time, the client data; and
storing, within a metadata database, one or more properties associated with the client data as a first set of entries associated with the first point-in-time;
perform, at a second point-in-time, an incremental backup of the client data to the object storage, the incremental backup including:
determining the client data stored on the client device at the second point-in-time includes new data since the first point-in-time;
storing, as a second set of objects associated with the second point-in-time, the new data, the second set of objects storing only the new data; and
storing, within the metadata database, properties associated with the new data as a second set of entries associated with the second point-in-time;
receive a request to perform a full restore of the client data to the second point-in-time;
determine the full restore requires retrieving data associated with the first and second points-in-time from the object storage in response to receiving the request to perform the full restore of the client data to the second point-in-time; and
combine, as part of performing the requested full restore, the first and second set of objects, based on determining the first and second set of objects are associated with the first and second points-in-time, with properties of the client data that were modified since the first point-in-time and stored in the metadata database.
2. The system of claim 1 , wherein the incremental backup further includes:
determining the client data stored on the client device at the second point-in-time further includes the modified data in which only the properties have changed since the first point-in-time; and
storing, within the metadata database, the changed properties associated with the modified data as a third set of entries associated with the second point-in-time, and
wherein the combining further includes combining the first and second set of objects with the third set of entries based on determining the third set of entries are associated with the second point-in-time.
3. The system of claim 2 , wherein combining the first and second set of objects with the third set of entries includes overwriting the properties of the client data stored as part of the first set of objects with the changed properties stored as part of the third set of entries.
4. The system of claim 1 , wherein the client data stored on the client device includes data files, and the properties associated with the client files includes metadata stored as part of the data files.
5. The system of claim 1 , wherein the client data includes client emails, and the properties associated with the client emails includes email properties.
6. The system of claim 1 , wherein storing, within the metadata database, the properties associated with the new data includes:
creating a new instance of the metadata database;
copying the first set of entries to the new instance; and
storing, within the new instance, the properties associated with the new data as the second set of entries associated with the second point-in-time.
7. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine data associated with the first point-in-time has expired; and
delete the first set of objects in response to determining the first set of objects are associated with the first point-in-time.
8. The system of claim 1 , wherein performing the full backup and the incremental backup includes accessing, by a server acting as an intermediary between the client device and the object storage, a storage application programming interface (API) of the object storage.
9. A method comprising:
performing, at a first point-in-time, a full backup of client data stored on a client device to a cloud-based object storage, the full backup including:
storing, as a first set of objects associated with the first point-in-time, the client data; and
storing, within a metadata database, one or more properties associated with the client data as a first set of entries associated with the first point-in-time;
performing, at a second point-in-time, an incremental backup of the client data to the object storage, the incremental backup including:
determining the client data stored on the client device at the second point-in-time includes new data since the first point-in-time;
storing, as a second set of objects associated with the second point-in-time, the new data, the second set of objects storing only the new data; and
storing, within the metadata database, the properties associated with the new data as a second set of entries associated with the second point-in-time;
receiving a request to perform a full restore of the client data to the second point-in-time;
determining the full restore requires retrieving data associated with the first and second points-in-time from the object storage in response to receiving the request to perform the full restore of the client data to the second point-in-time; and
combining, as part of performing the requested full restore, the first and second set of objects based on determining the first and second set of objects are associated with the first and second points-in-time, with properties of the client data that were modified since the first point-in-time and stored in the metadata database.
10. The method of claim 9 , wherein the incremental backup further includes:
determining the client data stored on the client device at the second point-in-time further includes the modified data in which only the properties have changed since the first point-in-time; and
storing, within the metadata database, the changed properties associated with the modified data as a third set of entries associated with the second point-in-time, and
wherein the combining further includes combining the first and second set of objects with the third set of entries based on determining the third set of entries are associated with the second point-in-time.
11. The method of claim 10 , wherein combining the first and second set of objects with the third set of entries includes overwriting the properties of the client data stored as part of the first set of objects with the changed properties stored as part of the third set of entries.
12. The method of claim 9 , wherein the client data stored on the client device includes data files, and the properties associated with the client files includes metadata stored as part of the data files.
13. The method of claim 9 , wherein the client data includes client emails, and the properties associated with the client emails includes email properties.
14. The method of claim 9 , wherein storing, within the metadata database, the properties associated with the new data includes:
creating a new instance of the metadata database;
copying the first set of entries to the new instance; and
storing, within the new instance, the properties associated with the new data as the second set of entries associated with the second point-in-time.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
perform, at a first point-in-time, a full backup of client data stored on a client device to a cloud-based object storage, the full backup including:
storing, as a first set of objects associated with the first point-in-time, the client data; and
storing, within a metadata database, one or more properties associated with the client data as a first set of entries associated with the first point-in-time;
perform, at a second point-in-time, an incremental backup of the client data to the object storage, the incremental backup including:
determining the client data stored on the client device at the second point-in-time includes new data since the first point-in-time;
storing, as a second set of objects associated with the second point-in-time, the new data, the second set of objects storing only the new data; and
storing, within the metadata database, the properties associated with the new data as a second set of entries associated with the second point-in-time;
receive a request to perform a full restore of the client data to the second point-in-time;
determine the full restore requires retrieving data associated with the first and second points-in-time from the object storage in response to receiving the request to perform the full restore of the client data to the second point-in-time; and
combine, as part of performing the requested full restore, the first and second set of objects based on determining the first and second set of objects are associated with the first and second points-in-time, with properties of the client data that were modified since the first point-in-time and stored in the metadata database.
16. The computer program product of claim 15 , wherein the incremental backup further includes:
determining the client data stored on the client device at the second point-in-time further includes the modified data in which only the properties have changed since the first point-in-time; and
storing, within the metadata database, the changed properties associated with the modified data as a third set of entries associated with the second point-in-time, and
wherein the combining further includes combining the first and second set of objects with the third set of entries based on determining the third set of entries are associated with the second point-in-time.
17. The computer program product of claim 16 , wherein combining the first and second set of objects with the third set of entries includes overwriting the properties of the client data stored as part of the first set of objects with the changed properties stored as part of the third set of entries.
18. The computer program product of claim 15 , wherein the client data stored on the client device includes data files, and the properties associated with the client files includes metadata stored as part of the data files.
19. The computer program product of claim 15 , wherein the client data includes client emails, and the properties associated with the client emails includes email properties.
20. The computer program product of claim 15 , wherein storing, within the metadata database, the properties associated with the new data includes:
creating a new instance of the metadata database;
copying the first set of entries to the new instance; and
storing, within the new instance, the properties associated with the new data as the second set of entries associated with the second point-in-time.