Method and apparatus to verify file metadata in a deduplication filesystem
One example method includes finding an Ln segment in a namespace, and adding the Ln segment to a fingerprint hashtable, scanning a container set and locating Ln in the container set, looking for Ln in the fingerprint hashtable, when Ln is found in the fingerprint hashtable, deleting Ln from the fingerprint hashtable, otherwise, retaining Ln in the fingerprint hashtable, when Ln is found in in the fingerprint hashtable, adding all fingerprints, pointed to by Ln, to the fingerprint hashtable, and when all levels of a tree that includes Ln have been checked, identifying as missing, any fingerprints still remaining in the fingerprint hashtable.
1 . A method, comprising:
repeating following operations by decreasing ‘n’ in Ln from 6 to 1:
finding a Ln metadata segment in a namespace, and adding the Ln metadata segment, which is a fingerprint, to a fingerprint hashtable;
scanning a container set and locating a Ln segment in the container set;
looking for a fingerprint corresponding to the Ln segment in the fingerprint hashtable;
when the fingerprint corresponding to the Ln segment is found in the fingerprint hashtable:
deleting the corresponding fingerprint from the fingerprint hashtable; and
adding all fingerprints, which are from the Ln segment corresponding to the deleted fingerprint, to the fingerprint hashtable; and
while adding the all fingerprints, if a duplicate fingerprint is found, not adding the duplicate fingerprint to the fingerprint hashtable; and
when all levels of a tree that includes Ln metadata segments have been checked, identifying, as missing, any fingerprints still remaining in the fingerprint hashtable.
2 . The method as recited in claim 1 , wherein each fingerprint is a hash of a respective data segment that resides at a bottom level of the tree.
3 . The method as recited in claim 1 , wherein the namespace spans a local storage tier and a cloud storage tier.
4 . The method as recited in claim 1 , wherein the missing fingerprints indicate that a data loss has occurred.
5 . The method as recited in claim 1 , wherein the levels of the tree are checked from a top of the tree to a bottom of the tree.
6 . The method as recited in claim 1 , further comprising using the missing fingerprints to identify a file that includes data segments to which the missing fingerprints correspond.
7 . The method as recited in claim 1 , wherein a top segment of the tree comprises a hash of an entire file, and the file is identified by walking the namespace and matching the file to the top segment.
8 . The method as recited in claim 7 , wherein the file is identified as missing one or more data segments.
9 . The method as recited in claim 1 , wherein there is a one-to-one mapping between a pair and a fingerprint index, and
a segment exists if there is a live reference to that segment.
10 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
repeating following operations by decreasing ‘n’ in Ln from 6 to 1:
finding a Ln metadata segment in a namespace, and adding the Ln metadata segment, which is a fingerprint, to a fingerprint hashtable;
scanning a container set and locating a Ln segment in the container set;
looking for a fingerprint corresponding to the Ln segment in the fingerprint hashtable;
when the fingerprint corresponding to the Ln segment is found in the fingerprint hashtable:
deleting the corresponding fingerprint from the fingerprint hashtable; and
adding all fingerprints, which are from the LN Ln segment corresponding to the deleted fingerprint, to the fingerprint hashtable; and
while adding the all fingerprints, if a duplicate fingerprint is found, not adding the duplicate fingerprint to the fingerprint hashtable; and
when all levels of a tree that includes Ln metadata segments have been checked, identifying, as missing, any fingerprints still remaining in the fingerprint hashtable.
11 . The non-transitory storage medium as recited in claim 10 , wherein each fingerprint is a hash of a respective data segment that resides at a bottom level of the tree.
12 . The non-transitory storage medium as recited in claim 10 , wherein the namespace spans a local storage tier and a cloud storage tier.
13 . The non-transitory storage medium as recited in claim 10 , wherein the missing fingerprints indicate that a data loss has occurred.
14 . The non-transitory storage medium as recited in claim 10 , wherein the levels of the tree are checked from a top of the tree to a bottom of the tree.
15 . The non-transitory storage medium as recited in claim 10 , wherein the operations further comprise using the missing fingerprints to identify a file that includes data segments to which the missing fingerprints correspond.
16 . The non-transitory storage medium as recited in claim 10 , wherein a top segment of the tree comprises a hash of an entire file, and the file is identified by walking the namespace and matching the file to the top segment.
17 . The non-transitory storage medium as recited in claim 16 , wherein the file is identified as missing one or more data segments.
18 . The non-transitory storage medium as recited in claim 10 , wherein there is a one-to-one mapping between a pair and a fingerprint index, and
a segment exists if there is a live reference to that segment.