Systems and methods for reading objects in a file system
View Patent ↗Embodiments of the disclosure relate generally to file systems. Specifically, certain embodiments include systems and methods for reading objects in a file system. In some embodiments, a first processing thread traverses a portion of a file system and represents file system objects in a data structure. A portion of the data and/or metadata associated with the represented objects may be prefetched to a memory. In some embodiments, a second processing thread consumes the objects represented in the queue. For example, in a file system backup embodiment, the second processing thread may transfer data and/or metadata associated with the objects to an archive target.
1. A computer system for transferring a portion of a file system to a memory, the system comprising:
a file system stored on at least one storage device;
one or more computer processors;
a memory coupled to the storage device;
a target stream;
a data structure representing files in the file system;
a prefetch module configured to, when executed by the one or more computer processors:
traverse data related to a portion of the file system;
determine whether to represent a file in the data structure based at least in part on a size of the file: a cumulative size of files currently represented in the data structure, wherein the cumulative size of the files represented in the a data structure comprises approximate cumulative space on the at least one storage device that is associated with the files, and a cumulative number of files currently represented in the data structure,
wherein the prefetch module determines not to represent the file in the data structure if to do so would cause the cumulative size of files to exceed a first threshold or would cause the cumulative number of files to exceed a second threshold,
wherein the first threshold and the second threshold are dynamically adjustable based on one or more of CPU usage, memory usage, transfer speed of a communication medium, and transfer speed of the at least one device; and
if the determination is to add a representation of the file to the data structure, add the representation of the file, prefetch at least a portion of the file, and add the prefetched portion to the memory; and
a working module configured to, when executed by the one or more computer processors, transfer at least portions of files represented in the data structure from the memory to the target stream.
2. The system of claim 1 , wherein the file system comprises a distributed file system and the at least one storage device comprises a plurality of storage devices.
3. The system of claim 1 , wherein the memory comprises a cache.
4. The system of claim 1 , wherein the data structure comprises a queue.
5. The system of claim 1 , wherein the portion of the file system comprises a directory.
6. The system of claim 1 , wherein the approximate cumulative space comprises approximate cumulative space of the files represented in the data structure that have not been transferred to the target stream by the working module.
7. The system of claim 1 , wherein the portion prefetched by the prefetch module comprises metadata associated with the file.
8. The system of claim 1 , wherein the prefetched portion comprises data associated with the file.
9. The system of claim 8 , wherein the data comprises only a beginning portion of the file.
10. The system of claim 1 , wherein the prefetch module is further configured such that the traversed portion of the file system is selected to be sufficient for the cumulative size of files represented in the data structure to change by at least a predetermined amount.
11. The system of claim 1 , wherein the working module is further configured to update the cumulative size of files represented in the data structure based at least in part on the size of the files transferred to the target stream.
12. The system of claim 1 , wherein the numerical count is weighted to reflect properties of the represented files.
13. The system of claim 9 , wherein the working module is further configured to retrieve a portion of the file which was not prefetched by the prefetch module.