Method to enhance the data invulnerability architecture of deduplication systems by optimally doing read-verify and fix of data moved to cloud tier
Data protection operations including verification operations are disclosed. Objects written to a cloud tier are verified without reading the objects out of the cloud. A translation map is used that allows a cloud verifier engine to compare a checksum of an object generated at an local tier with a checksum of the object as stored in the cloud tier. Mismatches are identified and corrective actions can be taken by reconstructing and rewriting the object to the cloud tier. Garbage collection may be prevented from reclaiming data associated with objects that have not been verified.
1. A method, comprising:
generating a list file that includes entries from a translation map, wherein each entry identifies a cloud object stored in a cloud storage, a first checksum of the cloud object, and segments stored in a local tier corresponding to the cloud object, wherein the first checksum was generated at the local tier;
verifying cloud objects associated with the entries in the list file by, for each entry:
reading a cloud object at the cloud storage;
generating a second checksum from the cloud object read from the cloud storage;
retrieving a third checksum from metadata of the cloud object;
comparing the first checksum, the second checksum and the third checksum, wherein the cloud object is validated when the first, second and third checksums match and wherein the cloud object is not validated when there is a mismatch between the first, second, and third checksums; and
logging the mismatch in a log file; and
processing the log file, by the local verifier engine, wherein entries in the translation map corresponding to objects not referenced in the log file or that are identified as verified are removed from the translation map.
2. The method of claim 1 , further comprising reconstructing cloud objects that were not verified.
3. The method of claim 1 , further comprising iterating the translation map.
4. The method of claim 1 , wherein the third checksum is generated before the cloud object is written to a disk.
5. The method of claim 1 , further comprising reconstructing objects in the log file from the local tier based on the segments identified in the translation map.
6. The method of claim 1 , further comprising performing garbage collection at the local tier, wherein data corresponding to objects identified in the translation map are not reclaimed during the garbage collection.
7. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
generating a list file that includes entries from a translation map, wherein each entry identifies a cloud object stored in a cloud storage, a first checksum of the cloud object, and segments stored in a local tier corresponding to the cloud object, wherein the first checksum was generated at the local tier;
verifying cloud objects associated with the entries in the list file by, for each entry:
reading a cloud object at the cloud storage;
generating a second checksum from the cloud object read from the cloud storage;
retrieving a third checksum from metadata of the cloud object;
comparing the first checksum, the second checksum and the third checksum, wherein the cloud object is validated when the first, second and third checksums match and wherein the cloud object is not validated when there is a mismatch between the first, second, and third checksums; and
logging the mismatch in a log file; and
processing the log file, by the local verifier engine, wherein entries in the translation map corresponding to objects not referenced in the log file or that are identified as verified are removed from the translation map.
8. The non-transitory storage medium of claim 7 , further comprising reconstructing cloud objects that were not verified.
9. The non-transitory storage medium of claim 7 , further comprising iterating the translation map.
10. The non-transitory storage medium of claim 7 , wherein the third checksum is generated before the cloud object is written to a disk.
11. The non-transitory storage medium of claim 7 , further comprising reconstructing objects in the log file from the local tier based on the segments identified in the translation map.
12. The non-transitory storage medium of claim 7 , further comprising performing garbage collection at the local tier, wherein data corresponding to objects identified in the translation map are not reclaimed during the garbage collection.
13. A method comprising:
verifying objects by: receiving objects at a cloud tier for storage in the cloud tier, wherein names of the objects are appended with corresponding checksums generated at a local tier;
after writing the objects to the cloud tier, performing for each object by a cloud verifier engine: extracting a first checksum from a name of the object; reading the object and generating a second checksum from the read object; comparing the first and second checksums; and logging the object in a log file when there is a mismatch between the first and second checksums;
renaming the objects when the first and second checksums match;
and processing the log file, by the local verifier engine, wherein entries in the translation map corresponding to objects not referenced in the log file or that are identified as verified are removed from the translation map.
14. The method of claim 13 , wherein writing the objects to the cloud tier triggers the cloud verifier engine.
15. The method of claim 14 , further comprising reconstructing the objects in the log file from segments stored in the local tier, wherein the reconstructed objects are added to a translation map.
16. The method of claim 15 , further comprising verifying the reconstructed objects after the reconstructed objects are committed to the cloud tier.
17. The method of claim 13 , further comprising verifying the objects in a parallel manner such that multiple objects are verified at the same time.
18. A non-transitory computer readable medium comprising computer executable instructions for performing the method of claim 13 .