Recovering the Metadata of Data Backed Up in Cloud Object Storage
Techniques for recovering metadata associated with data backed up in cloud object storage are provided. In one set of embodiments, a computer system can create a snapshot of a data set, where the snapshot includes a plurality of data blocks of the data set that have been modified since the creation of a prior snapshot of the data set. The computer system can further upload the snapshot to a cloud object storage platform of a cloud infrastructure, where the snapshot is uploaded as a plurality of log segments conforming to an object format of the cloud object storage platform, and where each log segment includes one or more data blocks in the plurality of data blocks, and a set of metadata comprising, for each of the one or more data blocks, an identifier of the data set, an identifier of the snapshot, and a logical block address (LBA) of the data block. The computer system can then communicate the set of metadata to a server component running in a cloud compute and block storage platform of the cloud infrastructure.
1 . A method comprising:
receiving, by a server that is part of a cloud compute and block storage platform of a cloud infrastructure, metadata pertaining to a segment of a data snapshot, wherein the metadata is received from an uploader agent of a source data center while the uploader agent is uploading the segment to a cloud object storage platform of the cloud infrastructure;
converting, by the server, the metadata into a set of metadata entries that conform to a schema of a metadata database of the cloud compute and block storage platform;
storing, by the server, the set of metadata entries in the metadata database; and
upon determining that there are further segments of the data snapshot to be uploaded to the cloud object storage platform:
returning, by the server, a first acknowledgement to the uploader agent that the metadata database has been updated with the metadata for the segment; and
after returning the first acknowledgement, replicating, by the server, changes in a transaction log of the metadata database caused by the storing of the metadata entries to a remote site.
2 . The method of claim 1 further comprising, upon determining that there are no further segments of the data snapshot to be uploaded to the cloud object storage platform:
replicating remaining changes in the transaction log to the remote site;
waiting for a second acknowledgement from the remote site that the replicating of the remaining changes is successful; and
in response to receiving the second acknowledgement, returning a third acknowledgement to the uploader agent indicating that upload of the data snapshot is complete.
3 . The method of claim 1 wherein, at a time of a failure at the cloud compute and block storage platform that causes contents of the metadata database to be lost, a metadata recovery agent of the cloud compute and block storage platform:
retrieves the transaction log from the remote site; and
rebuilds the metadata database by replaying the retrieved transaction log.
4 . The method of claim 1 wherein the metadata comprises user authentication information and an upload timestamp for the segment.
5 . The method of claim 1 wherein the replicating is performed by a background process of the server while the server receives and processes another segment of the data snapshot.
6 . The method of claim 1 wherein the segment is a fixed-size portion of the data snapshot that conforms to an object format supported by the cloud object storage platform.
7 . The method of claim 1 wherein the cloud compute and block storage platform provides a lower degree of storage durability than the cloud object storage platform.
8 . A non-transitory computer readable storage medium having stored thereon program code executable by a server that is part of a cloud compute and block storage platform of a cloud infrastructure, the program code embodying a method comprising:
receiving metadata pertaining to a segment of a data snapshot, wherein the metadata is received from an uploader agent of a source data center while the uploader agent is uploading the segment to a cloud object storage platform of the cloud infrastructure;
converting the metadata into a set of metadata entries that conform to a schema of a metadata database of the cloud compute and block storage platform;
storing the set of metadata entries in the metadata database; and
upon determining that there are further segments of the data snapshot to be uploaded to the cloud object storage platform:
returning a first acknowledgement to the uploader agent that the metadata database has been updated with the metadata for the segment; and
after returning the first acknowledgement, replicating changes in a transaction log of the metadata database caused by the storing of the metadata entries to a remote site.
9 . The non-transitory computer readable storage medium of claim 8 wherein the method further comprises, upon determining that there are no further segments of the data snapshot to be uploaded to the cloud object storage platform:
replicating remaining changes in the transaction log to the remote site;
waiting for a second acknowledgement from the remote site that the replicating of the remaining changes is successful; and
in response to receiving the second acknowledgement, returning a third acknowledgement to the uploader agent indicating that upload of the data snapshot is complete.
10 . The non-transitory computer readable storage medium of claim 8 wherein, at a time of a failure at the cloud compute and block storage platform that causes contents of the metadata database to be lost, a metadata recovery agent of the cloud compute and block storage platform:
retrieves the transaction log from the remote site; and
rebuilds the metadata database by replaying the retrieved transaction log.
11 . The non-transitory computer readable storage medium of claim 8 wherein the metadata comprises user authentication information and an upload timestamp for the segment.
12 . The non-transitory computer readable storage medium of claim 8 wherein the replicating is performed by a background process of the server while the server receives and processes another segment of the data snapshot.
13 . The non-transitory computer readable storage medium of claim 8 wherein the segment is a fixed-size portion of the data snapshot that conforms to an object format supported by the cloud object storage platform.
14 . The non-transitory computer readable storage medium of claim 8 wherein the cloud compute and block storage platform provides a lower degree of storage durability than the cloud object storage platform.
15 . A server that is part of a cloud compute and block storage platform of a cloud infrastructure, the server comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
receive metadata pertaining to a segment of a data snapshot, wherein the metadata is received from an uploader agent of a source data center while the uploader agent is uploading the segment to a cloud object storage platform of the cloud infrastructure;
convert the metadata into a set of metadata entries that conform to a schema of a metadata database of the cloud compute and block storage platform;
store the set of metadata entries in the metadata database; and
upon determining that there are further segments of the data snapshot to be uploaded to the cloud object storage platform:
return a first acknowledgement to the uploader agent that the metadata database has been updated with the metadata for the segment; and
after returning the first acknowledgement, replicate changes in a transaction log of the metadata database caused by the storing of the metadata entries to a remote site.
16 . The server of claim 15 wherein the program code further causes the processor to, upon determining that there are no further segments of the data snapshot to be uploaded to the cloud object storage platform:
replicate remaining changes in the transaction log to the remote site;
wait for a second acknowledgement from the remote site that the replicating of the remaining changes is successful; and
in response to receiving the second acknowledgement, return a third acknowledgement to the uploader agent indicating that upload of the data snapshot is complete.
17 . The server of claim 15 wherein, at a time of a failure at the cloud compute and block storage platform that causes contents of the metadata database to be lost, a metadata recovery agent of the cloud compute and block storage platform:
retrieves the transaction log from the remote site; and
rebuilds the metadata database by replaying the retrieved transaction log.
18 . The server of claim 15 wherein the metadata comprises user authentication information and an upload timestamp for the segment.
19 . The server of claim 15 wherein the processor performs the replicating via a background process while the processor receives and processes another segment of the data snapshot.
20 . The server of claim 15 wherein the segment is a fixed-size portion of the data snapshot that conforms to an object format supported by the cloud object storage platform.
21 . The server of claim 15 wherein the cloud compute and block storage platform provides a lower degree of storage durability than the cloud object storage platform.