Method or apparatus for scalable file verification in a deduplicated filesystem
One example method includes calculating a size of a fingerprint hashtable, calculating a number of verification passes that are needed, partitioning a fingerprint index table into index ranges corresponding to the number of verification passes needed, and during each verification pass for each index range: (1) finding all Ln segments in a namespace, and adding the Ln segments to the fingerprint hashtable and to a walk-vector data structure, (2) scanning a container set and locating any Ln segments in the container set; (3) reading all Ln-1 segments pointed to by the Ln segments and adding the Ln-1 segments to the fingerprint hashtable and to the walk-vector data structure; (4) repeating 1-3 for all levels of the tree structure that include Ln segments; and (5) comparing the fingerprint hashtable with the portion of the fingerprint index table corresponding to the index range to identify any missing Ln segments.
1 . A method, comprising:
calculating a size of a fingerprint hashtable based on a total number of Ln segments that are included in a namespace and based on available memory and on-disk space;
calculating a number of verification passes that are needed based on if the total number of Ln segments fit in the available on-disk space;
partitioning a fingerprint index table into a plurality of index ranges corresponding to the number of verification passes needed and determining a start and stop value for each of the index ranges, wherein when only one verification pass is needed, partitioning is not performed, and the fingerprint index table is considered as one index range that will be subjected to the one verification pass, wherein the steps of calculating a size of a fingerprint, calculating a number of verification passes, and partitioning a fingerprint index table occur before any verification pass is performed; and
during each verification pass for each index range of the plurality of index ranges of the fingerprint index:
(1) finding all Ln segments of a highest level of a tree structure in a namespace, and adding the Ln segments to the fingerprint hashtable and to a walk-vector data structure that is configured to keep track of Ln segments to read and Ln segments for which a read is completed, wherein the walk-vector data structure uses two bits for each segment, one bit indicating that the segment needs to be read, and one bit indicating that the segment has been read;
(2) scanning a container set and locating any Ln segments of the highest level of the tree structure in the container set;
(3) for all Ln segments of the highest level of the tree structure that are present in the walk-vector data structure, reading all Ln-1 segments pointed to by the Ln segments and adding the Ln-1 segments to the fingerprint hashtable and to the walk-vector data structure;
(4) repeating 1-3 for all levels of the tree structure below the highest level that include Ln segments, and if the fingerprint hashtable becomes full during the verification pass, writing the contents of the fingerprint hashtable to an on-disk file and continuing to add further segment fingerprints to an emptied fingerprint hashtable; and
(5) when all levels of the tree structure that include Ln segments have been checked, comparing the fingerprint hashtable with the portion of the fingerprint index table corresponding to the index range to identify any missing Ln segments.
2 . The method as recited in claim 1 , wherein a value of n ranges from 0 to 6, and the tree comprises a Merkle tree.
3 . The method as recited in claim 1 , wherein each fingerprint is a hash of a respective data segment or metadata segment.
4 . The method as recited in claim 1 , wherein the namespace spans a local storage tier and a cloud storage tier.
5 . The method as recited in claim 1 , wherein the missing fingerprints indicate that a data loss has occurred.
6 . The method as recited in claim 1 , wherein the levels of the tree are checked from a top of the tree to a bottom of the tree.
7 . The method as recited in claim 1 , further comprising using the missing fingerprints to identify a file that includes data segments to which the fingerprints correspond.
8 . The method as recited in claim 1 , wherein a top segment of the tree comprises a hash of an entire file, and the file is identified by walking the namespace and matching the file to the top segment.
9 . The method as recited in claim 8 , wherein the file is identified as missing one or more data segments.
10 . The method as recited in claim 1 , wherein there is a one-to-one mapping between a [segment, container] pair and the fingerprint index table, and a segment exists if there is a live reference to that segment.
11 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
calculating a size of a fingerprint hashtable based on a total number of Ln segments that are included in a namespace and based on available memory and on-disk space;
calculating a number of verification passes that are needed based on if the total number of Ln segments fit in the available on-disk space;
partitioning a fingerprint index table into a plurality of index ranges corresponding to the number of verification passes needed and determining a start and stop value for each of the index ranges, wherein when only one verification pass is needed, partitioning is not performed, and the fingerprint index table is considered as one index range that will be subjected to the one verification pass, wherein the steps of calculating a size of a fingerprint, calculating a number of verification passes, and partitioning a fingerprint index table occur before any verification pass is performed; and
during each verification pass for each index range of the plurality of index ranges of the fingerprint index:
(1) finding all Ln segments of a highest level of a tree structure in a namespace, and adding the Ln segments to the fingerprint hashtable and to a walk-vector data structure that is configured to keep track of Ln segments to read and Ln segments for which a read is completed, wherein the walk-vector data structure uses two bits for each segment, one bit indicating that the segment needs to be read, and one bit indicating that the segment has been read;
(2) scanning a container set and locating any Ln segments of the highest level of the tree structure in the container set;
(3) for all Ln segments of the highest level of the tree structure that are present in the walk-vector data structure, reading all Ln-1 segments pointed to by the Ln segments and adding the Ln-1 segments to the fingerprint hashtable and to the walk-vector data structure;
(4) repeating 1-3 for all levels of the tree structure below the highest level that include Ln segments, and if the fingerprint hashtable becomes full during the verification pass, writing the contents of the fingerprint hashtable to an on-disk file and continuing to add further segment fingerprints to an emptied fingerprint hashtable; and
(5) when all levels of the tree structure that include Ln segments have been checked, comparing the fingerprint hashtable with the portion of the fingerprint index table corresponding to the index range to identify any missing Ln segments.
12 . The non-transitory storage medium as recited in claim 11 , wherein a value of n ranges from 0 to 6, and the tree comprises a Merkle tree.
13 . The non-transitory storage medium as recited in claim 11 , wherein each fingerprint is a hash of a respective data segment or metadata segment.
14 . The non-transitory storage medium as recited in claim 11 , wherein the namespace spans a local storage tier and a cloud storage tier.
15 . The non-transitory storage medium as recited in claim 11 , wherein the missing fingerprints indicate that a data loss has occurred.
16 . The non-transitory storage medium as recited in claim 11 , wherein the levels of the tree are checked from a top of the tree to a bottom of the tree.
17 . The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise using the missing fingerprints to identify a file that includes data segments to which the fingerprints correspond.
18 . The non-transitory storage medium as recited in claim 11 , wherein a top segment of the tree comprises a hash of an entire file, and the file is identified by walking the namespace and matching the file to the top segment.
19 . The non-transitory storage medium as recited in claim 18 , wherein the file is identified as missing one or more data segments.
20 . The non-transitory storage medium as recited in claim 11 , wherein there is a one-to-one mapping between a [segment,container] pair and the fingerprint index table, and a segment exists if there is a live reference to that segment.