Systems and methods for providing a distributed file system utilizing metadata to track information about data stored throughout the system
View Patent ↗The intelligent distributed file system enables the storing of file data among a plurality of smart storage units which are accessed as a single file system. The intelligent distributed file system utilizes a metadata data structure to track and manage detailed information about each file, including, for example, the device and block locations of the file's data blocks, to permit different levels of replication and/or redundancy within a single file system, to facilitate the change of redundancy parameters, to provide high-level protection for metadata, to replicate and move data in real-time, and so forth.
1. A distributed file storage system comprising:
a plurality of storage units, comprising:
a first storage unit comprising a first processing module;
a second storage unit comprising a second processing module; and
a third storage unit comprising a third processing module, the third storage unit different from the first storage unit and the second storage unit;
wherein the first processing module, the second processing module, and the third processing module are configured to process read requests for files stored in a distributed manner across the plurality of storage units;
a file stored in a distributed manner across the plurality of storage units, the file comprising:
a first file portion stored in the first storage unit; and
a second file portion stored in the second storage unit, wherein the second file portion is different from the first file portion; and
file metadata comprising identifications of storage units in which the first file portion and the second file portion are stored and a reference to a parity map, the parity map comprising identifications of storage units in which parity blocks for the file are stored, at least a first portion of the file metadata stored in the third storage unit;
each of the first processing module, the second processing module, and the third processing module further configured to, in real-time:
receive the first portion of file metadata stored in the third storage unit;
receive the first file portion stored in the first storage unit and the second file portion stored in the second storage unit;
generate a parity block based on at least the first file portion and the second file portion;
select a fourth storage unit from the plurality of storage units on which to store the parity block based on one or more of a storage capacity, processor utilization and network utilization of one or more of the plurality of storage units, the fourth storage unit different from the first storage unit and the second storage unit to permit data recovery if the first storage unit or the second storage unit is unavailable;
store the parity block on the fourth storage unit; and
update the parity map to comprise an identification of the fourth storage unit.
2. The system of claim 1 , further comprising directory metadata to identify the location of the file metadata.
3. The system of claim 1 , further comprising a switch module in communication with the plurality of storage units, the switch module configured to receive a read request for the file stored on the distributed file storage system and to send the read request to any one of the plurality of storage units.
4. The system of claim 1 , wherein each of the plurality of storage units comprises at least one executable software module for flexibly distributing across the plurality of storage units at least one of the following: the file metadata, a copy of the file metadata, the first file portion, and the second file portion.
5. The system of claim 1 , wherein the file metadata further comprises identifications of blocks in memories of the storage units in which the first file portion and the second file portion are stored.
6. The system of claim 1 , further comprising mirrored file portions of the first file portion and the second file portion, and wherein the file metadata further comprises identifications of storage units in which the mirrored file portions are stored.
7. The system of claim 1 , wherein each of the first processing module, the second processing module, and the third processing module are further configured to:
monitor an access pattern for the file; and
automatically replicate or delete the file in response to a change in the access pattern.
8. The system of claim 2 , wherein each of the first processing module, the second processing module, and the third processing module are further configured to access the directory metadata, the directory metadata stored on a fifth storage unit.