Bulk file verification
Systems and methods for verifying files in bulk in a file system. When files are represented by a segment tree, the levels of the segment trees are walked by level such that that multiple files are verified at the same time in order to identify missing segments. Then, a bottom up scan is performed using the missing segments to identify the files corresponding to the missing segments. The missing files can then be handled by the file system.
1. A method for verifying files in bulk in a computing system, the method comprising:
performing a top down segment breadth walk in each of multiple segment trees to identify missing segments in a file system;
performing a bottom up segment breadth walk in each of the multiple segment trees to identify files corresponding to the missing segments; and
performing an action each of the identified files, wherein the action includes one of reinserting the missing files into the file system, reconstructing the missing files, or performing garbage collections on the missing segments.
2. The method of claim 1 , wherein each segment tree includes multiple levels, wherein a bottom level corresponds to data segments and wherein levels above the bottom level correspond to metadata segments.
3. The method of claim 2 , wherein the metadata segments of a particular level are mostly stored in the same container and metadata segments of different levels of the segment trees are mostly stored in different containers such that random IOs are reduced and such that the containers, during the top down segment breadth walk and the bottom up segment breadth walk can be read sequentially.
4. The method of claim 3 , wherein the segment tree includes a level L( 0 ) for the data segments and levels L( 1 )-L( 6 ) for metadata segments.
5. The method of claim 1 , wherein the top down segment breadth walk uses a walk vector, a match vector and a missing vector, wherein the match vector includes identifies obtained from segment references and the missing vector stores identifies of missing segments.
6. The method of claim 5 , further comprising walking a directory manager to add all top level references to the walk vector, wherein references identified during the top down segment breadth walk are included in the walk vector and the match vector.
7. The method of claim 6 , further comprising identifying comparing the match vector with an index and adding missing data segments to the missing vector.
8. The method of claim 7 , wherein the bottom up segment breadth walk is configured to identify segments whose identifiers are associated with the segments in the missing vector.
9. The method of claim 8 , further comprising walking a namespace to identify files corresponding to top level segments placed in the missing vector by the bottom up segment breadth walk, wherein filenames corresponding to the missing segments are identified.
10. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations for verifying files in bulk in a computing system, the operations comprising:
performing a top down segment breadth walk in each of multiple segment trees to identify missing segments in a file system;
performing a bottom up segment breadth walk in each of the multiple segment trees to identify files corresponding to the missing segments; and
performing an action each of the identified files, wherein the action includes one of reinserting the missing files into the file system, reconstructing the missing files, or performing garbage collections on the missing segments.
11. The non-transitory storage medium of claim 10 , wherein each segment tree includes multiple levels, wherein a bottom level corresponds to data segments and wherein levels above the bottom level correspond to metadata segments.
12. The non-transitory storage medium of claim 11 , wherein the metadata segments of a particular level are mostly stored in the same container and metadata segments of different levels of the segment trees are mostly stored in different containers such that random IOs are reduced and such that the containers, during the top down segment breadth walk and the bottom up segment breadth walk can be read sequentially.
13. The non-transitory storage medium of claim 12 , wherein the segment tree includes a level L( 0 ) for the data segments and levels L( 1 )-L( 6 ) for metadata segments.
14. The non-transitory storage medium of claim 10 , wherein the top down segment breadth walk uses a walk vector, a match vector and a missing vector, wherein the match vector includes identifies obtained from segment references and the missing vector stores identifies of missing segments.
15. The non-transitory storage medium of claim 14 , further comprising walking a directory manager to add all top level references to the walk vector, wherein references identified during the top down segment breadth walk are included in the walk vector and the match vector.
16. The non-transitory storage medium of claim 15 , further comprising identifying comparing the match vector with an index and adding missing data segments to the missing vector.
17. The non-transitory storage medium of claim 16 , wherein the bottom up segment breadth walk is configured to identify segments whose identifiers are associated with the segments in the missing vector.
18. The non-transitory storage medium of claim 17 , further comprising walking a namespace to identify files corresponding to top level segments placed in the missing vector by the bottom up segment breadth walk, wherein filenames corresponding to the missing segments are identified.