Systems and methods for a distributed file system with data recovery
View Patent ↗The intelligent distributed file system enables the storing of file data among a plurality of smart storage units which are accessed as a single file system. The intelligent distributed file system utilizes a metadata data structure to track and manage detailed information about each file, including, for example, the device and block locations of the file's data blocks, to permit different levels of replication and/or redundancy within a single file system, to facilitate the change of redundancy parameters, to provide high-level protection for metadata, to replicate and move data in real-time, and to permit the creation of virtual hot spares among the smart storage units without the need to idle any single smart storage unit in the intelligent distributed file system.
1. A distributed file storage system configured to provide data recovery comprising:
a plurality of storage modules in communication via a network, each of the plurality of storage modules configured to process at least one of read and write requests on behalf of the entire distributed storage system;
file metadata that identifies the location of data blocks comprising a file stored on the storage system, wherein the data blocks are distributed across at least some of the plurality of storage modules, and wherein the file metadata is stored on at least two, but not all of the plurality of storage modules and wherein the file metadata is stored on a different storage module than at least a portion of the directory metadata that references the location of the file;
directory metadata configured to reference the file metadata; and
a data recovery module configured to reconstruct portions of the file corresponding to data blocks on failed storage modules, wherein the reconstructed data blocks are stored in available data storage locations on the storage modules, and wherein the file metadata is updated to identify the new data storage locations.
2. The distributed file storage system of claim 1 , wherein the file is stored in a hierarchical file system.
3. The distributed file storage system of claim 1 , wherein the available data storage locations comprise a portion of a virtual hot spare.
4. The distributed file storage system of claim 3 , wherein the virtual hot spare includes at least one of the following: a per stripe virtual hot spare, a per file virtual hot spare, and a per data virtual hot spare.
5. A distributed file storage system configured to provide data recovery comprising:
a plurality of storage modules in communication via a network, each of the plurality of storage modules configured to process at least one of read and write requests on behalf of the entire distributed storage system;
file metadata that identifies the location of data blocks comprising a file stored on the storage system, wherein the data blocks are distributed across at least some of the plurality of storage modules, and wherein the file metadata is stored on at least two, but not all of the plurality of storage modules;
a data recovery module configured to reconstruct portions of the file corresponding to data blocks on failed storage modules, wherein the reconstructed data blocks are stored in available data storage locations on the storage modules, and wherein the file metadata is updated to identify the new data storage locations; and
an allocation module configured to flexibly allocate the data blocks and the file metadata across the storage modules of the storage system to improve the storage system's performance.
6. The distributed file storage system of claim 5 , wherein the file is stored in a hierarchical file system.
7. The distributed file storage system of claim 5 , further comprising directory metadata configured to reference the file metadata.
8. The distributed file storage system of claim 5 , wherein the allocation module further comprises changing the allocation of the data blocks and/or the file metadata across the storage modules of the storage system as the storage and/or retrieval demands change over time.
9. The distributed file storage system of claim 5 , wherein the available data storage locations comprise a portion of a virtual hot spare.
10. The distributed file storage system of claim 9 , wherein the virtual hot spare includes at least one of the following: a per stripe virtual hot spare, a per file virtual hot spare, and a per data virtual hot spare.