Specialized metadata for managing and searching backup data stored on a cloud-based object storage
Described is a system (and method) for managing specialized metadata that may be used to manage and search incremental backup data stored on a cloud-based object storage. The system may create and store such metadata as part of a specialized metadata database that includes a data catalog and a backup catalog. The system may leverage the metadata database to initiate operations to efficiently manage incremental backup data stored on the object storage. For example, the metadata may be relied upon to efficiently reconstruct (e.g. synthetically) the client data to a point-in-time of any incremental backup. In addition, the metadata may include properties of the backed-up data, which are maintained separately from the backup data stored as objects. Accordingly, these properties may be searched to identify and locate backup data without having to retrieve the stored objects.
1. A system comprising:
one or more processors; and
a non-transitory computer-readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
perform a backup of client data stored on a client device to a new set of objects stored on a cloud-based object storage, the client data including one or more data items each associated with one or more properties that are stored as part of the data item, the cloud-based object storage including existing objects associated with previous backups of existing client data;
obtain, for each of the data items, the associated properties;
store the associated properties as part of a set of entries within a data catalog, the associated properties for each data item stored as part of a corresponding entry that stores metadata of the data item;
receive a request to identify a first data item stored within the new set of objects, the request including a search value;
search the set of entries within the data catalog data using the search value;
identify a first entry, amongst the set of entries within the data catalog, corresponding to the first data item based on the search value matching the metadata stored by the first entry; and
retrieve the first data item from a first object, amongst the new set of objects, based on the metadata stored by the first entry indicating the first data item is stored within the first object.
2. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
store backup metadata associated with the backup as an entry within a backup catalog, the backup metadata including at least a backup time of the backup.
3. The system of claim 2 , wherein the metadata stored by the corresponding entries of the data items includes the backup time of the backup.
4. The system of claim 1 , wherein the backup includes an incremental backup from a previously performed backup, and obtaining the associated properties includes obtaining changes to the properties since the previously performed backup for at least some of the data items.
5. The system of claim 1 , wherein searching the set of entries within the data catalog data includes searching only the data catalog without accessing the new set of objects.
6. The system of claim 1 , wherein each data item includes an email, and the one or more properties include at least one of an email subject, sender, receipt, and sent time.
7. The system of claim 1 , wherein retrieving the first data item from the first object of the new set of objects identified by the first entry includes performing a full restore of the data items stored as part of the backup.
8. A method comprising:
performing a backup of client data stored on a client device to a new set of objects stored on a cloud-based object storage, the client data including one or more data items each associated with one or more properties that are stored as part of the data item, the cloud-based object storage including existing objects associated with previous backups of existing client data;
obtaining, for each of the data items, the associated properties;
storing the associated properties as part of a set of entries within a data catalog, the associated properties for each data item stored as part of a corresponding entry that stores metadata of the data item;
receiving a request to identify a first data item stored within the new set of objects, the request including a search value;
searching the set of entries within the data catalog data using the search value;
identifying a first entry, amongst the set of entries within the data catalog, corresponding to the first data item based on the search value matching the metadata stored by the first entry; and
retrieving the first data item from a first object, amongst the new set of objects, based on the metadata stored by the first entry indicating the first data item is stored within the first object.
9. The method of claim 8 , further comprising:
storing backup metadata associated with the backup as an entry within a backup catalog, the backup metadata including at least a backup time of the backup.
10. The method of claim 9 , wherein the metadata stored by the corresponding entries of the data items includes the backup time of the backup.
11. The method of claim 8 , wherein the backup includes an incremental backup from a previously performed backup, and obtaining the associated properties includes obtaining changes to the properties since the previously performed backup for at least some of the data items.
12. The method of claim 8 , wherein searching the set of entries within the data catalog data includes searching only the data catalog without accessing the new set of objects.
13. The method of claim 8 , wherein each data item includes an email, and the one or more properties include at least one of an email subject, sender, receipt, and sent time.
14. The method of claim 8 , wherein retrieving the first data item from the first object of the new set of objects identified by the first entry includes performing a full restore of the data items stored as part of the backup.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
perform a backup of client data stored on a client device to a new set of objects stored on a cloud-based object storage, the client data including one or more data items each associated with one or more properties that are stored as part of the data item, the cloud-based object storage including existing objects associated with previous backups of existing client data;
obtain, for each of the data items, the associated properties;
store the associated properties as part of a set of entries within a data catalog, the associated properties for each data item stored as part of a corresponding entry that stores metadata of the data item;
receive a request to identify a first data item stored within the new set of objects, the request including a search value;
search the set of entries within the data catalog data using the search value;
identify a first entry, amongst the set of entries within the data catalog, corresponding to the first data item based on the search value matching the metadata stored by the first entry; and
retrieve the first data item from a first object, amongst the new set of objects, based on the metadata stored by the first entry indicating the first data item is stored within the first object.
16. The computer program product of claim 15 , wherein the program code includes further instructions to:
store backup metadata associated with the backup as an entry within a backup catalog, the backup metadata including at least a backup time of the backup.
17. The computer program product of claim 16 , wherein the metadata stored by the corresponding entries of the data items includes the backup time of the backup.
18. The computer program product of claim 15 , wherein the backup includes an incremental backup from a previously performed backup, and obtaining the associated properties includes obtaining changes to the properties since the previously performed backup for at least some of the data items.
19. The computer program product of claim 15 , wherein searching the set of entries within the data catalog data includes searching only the data catalog without accessing the new set of objects.
20. The computer program product of claim 15 , wherein each data item includes an email, and the one or more properties include at least one of an email subject, sender, receipt, and sent time.