Improving the recovery time objective of backup recovery from the cloud
Embodiments for performing data recovery from cloud storage to a backup appliance that goes from an offline state to an online state, including identifying super metadata among metadata stored in the cloud storage for the backup appliance, wherein the super metadata comprises an amount on the order of ten percent of the metadata representing a minimum amount of information to reconstruct the metadata, copying the super metadata blocks from the cloud storage to local metadata storage of the backup appliance while it is in the offline state to leave remaining metadata, and copying the remaining metadata blocks from the cloud storage to the local metadata storage while the backup appliance is in the online state.
1. A computer-implemented method of performing data recovery from cloud storage to a file system server comprising a backup appliance that goes from an offline state to an online state, comprising:
identifying super metadata among metadata stored in the cloud storage for the file system server, wherein the super metadata comprises an amount on the order of ten percent or less of the metadata representing a minimum amount of information to reconstruct the metadata;
copying the super metadata blocks from the cloud storage to local metadata storage of the file system server while it is in the offline state to leave remaining metadata; and
copying the remaining metadata blocks from the cloud storage to the local metadata storage while the file system server is in the online state, wherein the remaining metadata blocks are copied as bulk copies copied in a background process, and further wherein the super metadata is recovered first from a mirrored copy on the storage before a filesystem of the file system server is made operational, wherein the super metadata comprises a data structure including an allocation table, a fingerprint index table, and a namespace tree.
2. The method of claim 1 wherein the cloud storage comprises storage media resident in a cloud computing network maintained by a cloud service provider, and provided for long term retention of data processed using the metadata.
3. The method of claim 1 wherein the metadata other than the super metadata comprises index node (inode) trees of the metadata and actual metadata blocks.
4. The method of claim 1 further comprising:
spawning a background thread to scrub the allocation table and copy the metadata blocks in a recovery pending state back to the local metadata storage; and
updating the recovery pending state as ‘allocated’ after copying the metadata blocks from the recovery pending state.
5. The method of claim 1 wherein the backup appliance comprises part of a deduplication backup system executed by a data storage server running a Data Domain file system (DDFS).
6. The method of claim 5 wherein the metadata is stored in a segment tree, and wherein leaf nodes in the segment tree comprise L0 segments or data segments, and the segment tree comprises the metadata (Lp) segments.
7. The method of claim 5 wherein the allocation table stores a state for each of the allocated blocks or chunks in the filesystem and represents whether a block is allocated or not, the fingerprint index table comprises a mapping of the block fingerprints and the actual block numbers wherein each block could either be a fixed size or variable size depending on a deduplication method, and wherein the namespace tree comprises directory hierarchies representing file paths.
8. A system comprising:
a backup appliance including local backup storage and executing a backup manager and storing data and associated metadata in an active tier and a cloud tier, wherein the cloud tier comprises storage media resident in a cloud computing network maintained by a cloud service provider, and provided for long term retention of data including the file, and the active tier comprises the local backup storage of the backup appliance; and
a cloud network storing data as for the backup appliance as cloud tier data and storing the metadata as cloud tier metadata, wherein a process of the backup appliance identifies super metadata among metadata stored in the cloud storage, the super metadata representing a minimum amount of information to reconstruct the metadata, copies the super metadata blocks from the cloud tier to the active tier while the backup appliance is offline to leave remaining metadata, and copies the remaining metadata blocks from the cloud tier to the active tier while the backup appliance is online, wherein the backup appliance comprises part of a deduplication backup system executed by a data storage server running a Data Domain file system (DDFS), and further wherein the super metadata comprises a data structure including an allocation table, a fingerprint index table, and a namespace tree.
9. The system of claim 8 wherein the allocation table stores a state for each of the allocated blocks or chunks in the filesystem and represents whether a block is allocated or not, the fingerprint index table comprises a mapping of the block fingerprints and the actual block numbers wherein each block could either be a fixed size or variable size depending on a deduplication method, and wherein the namespace tree comprises directory hierarchies representing file paths.
10. The system of claim 8 wherein the metadata other than the super metadata comprises actual metadata blocks and index node (inode) trees.
11. The system of claim 10 wherein the metadata is stored in a segment tree, and wherein leaf nodes in the segment tree comprise L0 segments or data segments, and the segment tree comprises the metadata (Lp) segments.
12. A method comprising:
configuring, in the event of a failure, a new backup appliance identical to an original backup appliance, the original backup appliance maintaining copies of data and associated metadata for one or more data sources in cloud storage;
copying, from the cloud storage, metadata blocks from the cloud storage to local storage of the new backup appliance;
identifying, in a pre-recovery phase, super metadata representing minimum metadata to reconstruct the metadata on the new backup appliance to leave remaining metadata, wherein the super metadata comprises a data structure including an allocation table, a fingerprint index table, and a namespace tree, and wherein user access comprises writing new data, accessing files, and listing files;
copying the super metadata from the cloud to the new backup appliance while the original backup appliance is offline and not available for user access due to the failure; and
copying, as a background task or as an on-demand process, the remaining metadata from the cloud to the new backup appliance while the new backup appliance is online and the user can access the backup appliance as if data recovery is complete.
13. The method of claim 12 wherein the remaining metadata comprises actual metadata blocks and index node (inode) trees.