Systems and methods for providing a distributed file system utilizing metadata to track information about data stored throughout the system
View Patent ↗The intelligent distributed file system enables the storing of file data among a plurality of smart storage units which are accessed as a single file system. The intelligent distributed file system utilizes a metadata data structure to track and manage detailed information about each file, including, for example, the device and block locations of the file's data blocks, to permit different levels of replication and/or redundancy within a single file system, to facilitate the change of redundancy parameters, to provide high-level protection for metadata, to replicate and move data in real-time, and so forth.
1. A distributed file storage system comprising:
a plurality of storage devices comprising at least a first storage device, a second storage device, a third storage device, and a fourth storage device, wherein each of the plurality of storage devices comprises a computer processor and at least one executable software module, wherein the first storage device, the second storage device, the third storage device and the fourth storage device are each a different storage device;
a file comprising multiple data blocks, the multiple data blocks comprising at least a first data block and a second data block, the first data block stored on the first storage device and the second data block stored on the second storage device;
file metadata comprising address locations of the first data block and the second data block, the file metadata stored on the first storage device; and
directory metadata comprising an address location of the file metadata, the directory metadata comprising:
a first portion of the directory metadata, wherein the first portion of the directory metadata is stored on the third storage device; and
a second portion of the directory metadata, wherein the second portion of the directory metadata is stored on the fourth storage device;
wherein the at least one executable software module of the first storage device is executed by a computer processor and configured to:
receive a request for the file; and
in response to the request:
query the third storage device for directory metadata;
receive the first portion of the directory metadata from the third storage device;
query the fourth storage device for directory metadata;
receive the second portion of the directory metadata from the fourth storage device;
retrieve the file metadata stored on the third first storage device to obtain the address locations of the first data block stored on the first storage device and the second data block stored on the second storage device;
retrieve the first data block stored on the first storage device; and
send a request to the second storage device for the second data block stored on the second storage device.
2. The system of claim 1 , wherein the address locations of the first data block and the second data block each comprise device information and block information.
3. The system of claim 1 , wherein the first data block is stored in a first memory on the first storage device, the second data block is stored in a second memory on the second storage device, and the size of the first memory is different from the size of the second memory.
4. The system of claim 1 , wherein the file metadata comprises a first data location table, the first data location table comprising the address location of the first data block and a reference to a second data location table, the second data location table comprising the address location of the second data block.
5. The system of claim 1 , wherein each of the first storage device, the second storage device, the third storage device, and the fourth storage device are configured to monitor an access pattern for the file, and the at least one executable software module of the first storage device is further configured to automatically replicate or delete the file in response to a change in the access pattern.
6. The system of claim 1 , wherein the plurality of storage devices are configured to communicate with each other via a first connection and a second connection, and the system is configured to transfer data blocks via the first connection and control messages via the second connection.
7. The system of claim 1 , wherein the at least one executable software module of the first storage device is further configured to return the first data block stored on the first storage device and the second data block stored on the second storage device.
8. The system of claim 1 , wherein the at least one executable software module of the first storage device is further configured to return the first data block stored on the first storage device and the second data block stored on the second storage device after all of the multiple data blocks of the file have been accessed.
9. The system of claim 1 , wherein the file metadata comprises a data location table comprising address locations of the multiple blocks of the file, and the at least one executable software module of each of the first storage device is further configured to return the first data block stored on the first storage device and the second data block stored on the second storage device in sequential order according to the data location table.
10. A method for processing a file request for a file stored by a distributed file storage system comprising a plurality of storage devices, the method comprising:
receiving, by a first storage device in a distributed file storage system, a request for a file comprising multiple data blocks, the distributed file storage system comprising at least the first storage device, a second storage device, a third storage device, and a fourth storage device, the multiple data blocks comprising at least a first data block and a second data block, the first data block stored on the first storage device and the second data block stored on the second storage device;
querying the third storage device for directory metadata;
receiving a first portion of directory metadata from the third storage device, the directory metadata comprising an address location of file metadata stored on the first storage device;
querying the fourth storage device for directory metadata;
receiving a second portion of the directory metadata from the fourth storage device;
retrieving from the first storage device the file metadata to obtain the address locations of the first data block stored on the first storage device and the second data block stored on the second storage device;
retrieving the first data block stored on the first storage device; and
sending a request to the second storage device for the second data block stored on the second storage device.
11. The method of claim 10 , wherein the address locations of the first data block and the second data block each comprise device information and block information.
12. The method of claim 10 , wherein the first data block is stored in a first memory on the first storage device and the second data block is stored in a second memory on the second storage device, wherein the size of the first memory is different from the size of the second memory.
13. The method of claim 10 , wherein the file metadata comprises a first data location table, the first data location table comprising the address location of the first data block and a reference to a second data location table, the second data location table comprising the address location of the second data block.
14. The method of claim 10 , further comprising:
monitoring, by at least one of the first storage device, the second storage device, the third storage device, and the fourth storage device, an access pattern for the file; and
automatically replicating or deleting the file in response to a change in the access pattern.
15. The method of claim 10 , wherein the first storage device, the second storage device, the third storage device, and the fourth storage device communicate with each other via a first connection and a second connection, the method further comprising:
transferring data blocks via the first connection; and
transferring control messages via the second connection.
16. The method of claim 10 , further comprising returning the first data block stored on the first storage device and the second data block stored on the second storage device.
17. The method of claim 10 , further comprising returning the first data block stored on the first storage device and the second data block stored on the second storage device after all of the multiple data blocks of the file have been accessed.
18. The method of claim 10 , wherein the file metadata comprises a data location table comprising address locations of the multiple blocks of the file, the method further comprising returning the first data block stored on the first storage device and the second data block stored on the second storage device in sequential order according to the data location table.