Distributed Storage System Data Management And Security
A system and method for distributing data over a plurality of remote storage nodes. Data are split into segments and each segment is encoded into a number of codeword chunks. None of the codeword chunks contains any of the segments. Each codeword chunk is packaged with at least one encoding parameter and identifier, and metadata are generated for at least one file and for related segments of the at least one file. The metadata contains information to reconstruct from the segments, and information for reconstructing from corresponding packages. Further, metadata are encoded into package(s), and correspond to a respective security level and a protection against storage node failure. A plurality of packages are assigned to remote storage nodes to optimize workload distribution. Each package is transmitted to at least one respective storage node as a function iteratively accessing and retrieving the packages of metadata and file data.
1 - 18 . (canceled)
19 . A method of data retrieval from remote storage nodes, the method comprising:
accessing, by one or more processors configured to execute code stored in non-transitory processor readable media, file metadata references within a local cache or within remote storage nodes;
receiving, by the one or more processors, a plurality of packages from remote storage nodes by metadata references, each of the packages contain file metadata;
receiving, by the one or more processors, a plurality of other packages containing encoded file segments from storage nodes by data references, wherein the encoded file segments are obtained at least partly from file metadata;
reconstructing, by the one or more processors, file data from the packages as a function of metadata representing parameters associated with an encoding scheme and file splitting scheme.
20 . The method of claim 19 , wherein file retrieval speed is enhanced by caching metadata from a plurality of client side files.
21 - 40 . (canceled)
41 . A system for data retrieval from remote storage nodes, the system comprising:
one or more processors in electronic communication with non-transitory processor readable storage media; and
one or more software modules comprising executable instructions stored in the storage media, wherein the one or more software modules are executable by the one or more processors that, when so executed, cause the one or more processors to:
access, by one or more processors, file metadata references within a local cache or within remote storage nodes;
receive, by the one or more processors, a plurality of packages from remote storage nodes by metadata references, each of the packages contain file metadata;
receive, by the one or more processors, a plurality of other packages containing encoded file segments from storage nodes by data references, wherein the encoded file segments are obtained at least partly from file metadata; and
reconstruct, by the one or more processors, file data from the packages as a function of metadata representing parameters associated with an encoding scheme and file splitting scheme.
42 . The system of claim 41 , wherein file retrieval speed is enhanced by caching metadata from a plurality of client side files.