Reconstructing lost data objects by generating virtual user files from available tiers within a node
Embodiments of method and system are directed to a best effort method to reconstruct the corrupt files from available nodes or tiers and heal or repair the corrupt files. The method looks for the existence of fingerprints of the lost segments in available tiers on the nodes or across nodes within a cluster. The method discovers the copies of these missing segments, creates an Lp tree referring these discovered segments and creates a virtual file in the namespace. The file is then moved to the destination tier where the corruption exists, thus healing the corruption.
1. A computer-implemented method of reconstructing and repairing corrupt files between an active tier and a cloud tier within a node comprising:
scanning container metadata (CMETA) containers to identify containers having references to missing cloud objects of a corrupted file;
looking for fingerprints of missing segments of the corrupted file in the cloud tier, wherein a missing segment comprises a data object missing in the cloud tier but still referenced by an L0 fingerprint;
discovering copies of these missing segments through an index lookup of missing fingerprints on other available nodes;
creating an Lp tree referring the discovered missing segments;
creating a temporary virtual file in a namespace encompassing both the active tier and the cloud tier;
rewriting an original CMETA container having the references to the missing cloud objects by dropping corresponding metadata sections of the missing cloud objects to remove entries of missing fingerprints from the index; and
moving the virtual file from the active tier to the cloud tier to thereby restore the missing segment and reconstruct the corrupted file.
2. The method of claim 1 wherein the cloud storage tier comprises storage media resident in a cloud computing network maintained by a cloud service provider, and provided for long term retention of data including the corrupted file.
3. The method of claim 2 wherein the active tier comprises storage media resident or closely coupled to a server computer of the node and executing one or more data processing operations on the file.
4. The method of claim 3 wherein content of the file comprises a lower segment of a Merkle tree, and wherein leaf nodes in the segment tree comprise L0 segments or data segments, and the segment tree comprises the metadata (Lp) segments.
5. The method of claim 4 wherein the fingerprint is one of a plurality of fingerprints for the file content and that are collected in Ln segments.
6. The method of claim 5 further comprising:
storing metadata of files including the corrupted file in a metadata container in local storage of the node;
scanning the metadata container to identify references to the missing segment; and
discovering the segment corresponding to a fingerprint using an index lookup of the index.
7. The method of claim 6 wherein the step of creating the temporary virtual file comprises generating a content handle representing L6 and stitching into the namespace.
8. The method of claim 1 further comprising updating a file migration policy and moving the temporary virtual file from the active tier to the cloud tier.
9. The method of claim 1 wherein the node comprises part of a deduplication backup process executed by a data storage server running a Data Domain file system (DDFS).
10. A system for reconstructing and repairing corrupt files between an active tier and a cloud tier within a node in a processor-based hardware circuit, comprising:
a processor-based first corrupt file repair component configured to scan container metadata (CMETA) containers to identify containers having references to missing cloud objects of a corrupted file, and look for fingerprints of missing segments of a corrupted file in the cloud tier wherein a missing segment comprises a data object missing in the cloud tier but still referenced by an L0 fingerprint, discover copies of these missing segments through an index lookup of missing fingerprints on other available nodes;
a processor-based second corrupt file repair component configured to create an Lp tree referring the discovered missing segments, and create a temporary virtual file in a namespace encompassing both the active tier and the cloud tier, and rewrite an original CMETA container having the references to the missing cloud objects by dropping corresponding metadata sections of the missing cloud objects to remove entries of missing fingerprints from the index; and
a hardware interface moving the virtual file from the active tier to the cloud tier to thereby restore the missing segment and reconstruct the corrupted file.
11. The system of claim 10 wherein the cloud storage tier comprises storage media resident in a cloud computing network maintained by a cloud service provider, and provided for long term retention of data including the corrupted file.
12. The system of claim 11 wherein the active tier comprises storage media resident or closely coupled to a server computer of the node and executing one or more data processing operations on the file.
13. The system of claim 12 wherein content of the file comprises a lower segment of a Merkle tree, and wherein leaf nodes in the segment tree comprise L0 segments or data segments, and the segment tree comprises the metadata (Lp) segments.
14. The system of claim 13 wherein the fingerprint is one of a plurality of fingerprints for the file content and that are collected in Ln segments.
15. The system of claim 14 further comprising a storage media storing metadata of files including the corrupted file in a metadata container in local storage of the node, the first corrupt file repair component further configured to scan the metadata container to identify references to the missing segment, and discover the segment corresponding to a fingerprint using an index lookup.
16. The system of claim 15 wherein creating the temporary virtual file comprises generating a content handle representing L6 and stitching into the namespace.
17. The system of claim 16 wherein the first corrupt file repair component is further configured to update a file migration policy and moving the temporary virtual file from the active tier to the cloud tier.
18. The system of claim 10 wherein the node comprises part of a deduplication backup process executed by a data storage server running a Data Domain file system (DDFS).
19. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method to reconstruct and repair corrupt files between an active tier and a cloud tier within a node by:
scanning container metadata (CMETA) containers to identify containers having references to missing cloud objects of a corrupted file;
looking for fingerprints of missing segments of the corrupted file in the cloud tier, wherein a missing segment comprises a data object missing in the cloud tier but still referenced by an L0 fingerprint;
discovering copies of these missing segments in the node through an index lookup of missing fingerprints on other available nodes;
creating an Lp tree referring the discovered missing segments;
creating a temporary virtual file in a namespace encompassing both the active tier and the cloud tier;
rewriting an original CMETA container having the references to the missing cloud objects by dropping corresponding metadata sections of the missing cloud objects to remove entries of missing fingerprints from the index; and
moving the virtual file from the active tier to the cloud tier to thereby restore the missing segment and reconstruct the corrupted file.