Recursive multi-threaded file system scanner for serializing file system metadata exoskeleton
Methods, apparatus, and other embodiments facilitate discovering and serializing metadata for a file system. After initializing or accessing a serialization data structure, recursive search threads that begin at a selected location in the file system are spawned. The threads analyzes the selected location in the file system, acquire metadata associated with the selected location in the file system, serialize the metadata associated with the selected location in the file system into the serialization data structure, and then selectively spawn additional recursive search threads. Additional recursive search threads may be spawned upon determining that the current recursive search thread has reached a branch point in a hierarchy of the file system. Different threads may perform different types of searches (e.g., depth-first breadth-first) and may operate in parallel. The serialization data structure may be persisted and surfaced in an object (e.g., JSON object) that can be queried or searched.
1. A non-transitory computer-readable storage device storing computer executable instructions that when executed by a computer control the computer to perform a method for serializing metadata associated with a file system, the method comprising:
accessing the file system, where the file system is a shared disk file system or a file locking storage area network file system;
initializing a serialization data structure;
spawning a current recursive search thread that begins at a selected location in the file system, where the current recursive search thread:
analyzes the selected location in the file system,
acquires metadata associated with the selected location in the file system,
serializes the metadata associated with the selected location in the file system into the serialization data structure, and
selectively spawns an additional recursive search thread upon determining that the current recursive search thread has reached a branch point in a hierarchy of the file system, where the additional recursive search thread begins at a selected node in a branch path available at the branch point.
2. The non-transitory computer-readable storage device of claim 1 , comprising persisting the serialization data structure in an object.
3. The non-transitory computer-readable storage device of claim 2 , where the object is a Javascript object notation (JSON) object.
4. The non-transitory computer-readable storage device of claim 1 , where the current recursive search thread is a depth-first search thread or a breadth-first search thread and where the additional recursive search thread is a depth-first search thread or a breadth-first search thread.
5. The non-transitory computer-readable storage device of claim 1 , comprising selecting whether the additional recursive search thread is a depth-first search thread or a breadth-first search thread based, at least in part, on metadata associated with the selected location in the file system.
6. The non-transitory computer-readable storage device of claim 1 , comprising selecting whether the additional recursive search thread is a depth-first search thread or a breadth-first search thread based, at least in part, on metadata associated with the selected node in the branch path available at the branch point.
7. The non-transitory computer-readable storage device of claim 1 , where the file system is a shared disk file system that provides filing locking and a single namespace for the file system in a storage area network configuration.
8. The non-transitory computer-readable storage device of claim 1 , where the metadata includes hierarchy information or file information.
9. The non-transitory computer-readable storage device of claim 2 , comprising:
accessing a query concerning the file system, and
satisfying the query from information in the object without accessing the file system.
10. The non-transitory computer-readable storage device of claim 2 , comprising:
accessing the object, and
rebuilding a portion of the file system from the object using a recursive, multi-threaded builder.
11. The non-transitory computer-readable storage device of claim 10 , where the recursive multi-threaded builder:
spawns a current recursive build thread that begins at a first location in the file system,
acquires metadata associated with the first location in the file system from the object,
populates the first location in the file system with the metadata associated with the first location in the file system, and
selectively spawns an additional recursive build thread upon determining that the current recursive build thread has reached a build branch point in the hierarchy of the file system, where the additional recursive build thread begins at a second node in a branch path associated with the build branch point.
12. An apparatus for serializing metadata associated with a file system, comprising:
a processor;
a memory that stores information concerning a multi-threaded traversal of the file system, where the file system is a shared disk file system or a file locking storage area network file system;
a set of circuits; and
an interface to connect the processor, the memory, and the set of circuits,
the set of circuits comprising:
a first circuit configured to spawn a recursive thread that is configured to:
acquire metadata for locations in the file system visited by the recursive thread, and
make a request to the first circuit to spawn an additional recursive thread at a branch point in a hierarchy of the file system,
a second circuit configured to serialize metadata acquired by recursive threads spawned by the first circuit into an object, and
a third circuit configured to persist and surface the object.
13. The apparatus of claim 12 , comprising:
a fourth circuit configured to build a copy of the file system from the object using a recursive tree building thread.
14. The apparatus of claim 12 , comprising a fifth circuit configured to satisfy a query concerning the file system metadata associated with the file system from the object.
15. The apparatus of claim 12 , where the recursive thread is a depth-first search thread and where the additional recursive thread is a depth-first search thread.
16. The apparatus of claim 12 , where the recursive thread is a breadth-first search thread and where the additional recursive thread is a breadth-first search thread.
17. The apparatus of claim 12 , where the recursive thread is a depth-first search thread or a breadth-first search thread and where the additional recursive thread is a depth-first search thread or a breadth-first search thread, and where the first circuit chooses whether to spawn a depth-first search thread or a breadth-first search thread based, at least in part, on metadata associated with the branch point in the hierarchy of the file system.