Distributed storage system data management and security
Secure distributed storage and transmission of electronic content is provided over at least one communication network. At least one data file is received and parsed into a plurality of segments, wherein each one of the segments has a respective size. Thereafter, each of the plurality of segments is divided into a plurality of slices, wherein each one of the slices has a respective size. A plurality of data chunks are encoded, each data chunk comprising a portion of at least two of the slices, wherein no portion comprises an entire slice. The data chunks are packaged with at least metadata, and each of the packages is assigned to respective remote storage nodes. Each of the packages is transmitted to the respectively assigned remote storage node.
1. A method for providing secure distributed storage and transmission of electronic content over at least one communication network, the method comprising:
receiving, by at least one processor configured by executing instructions stored on non-transitory processor-readable media, at least one data file;
parsing, by the at least one processor, the at least one data file into a plurality of segments, wherein each one of the segments has a respective size;
dividing, by the at least one processor, each of the plurality of segments into a plurality of slices, wherein each one of the slices has a respective size;
encoding, by the at least one processor, a plurality of data chunks, each data chunk comprising i) a portion of at least two of the slices from a first segment and ii) a plurality of portions of at least one respective slice from each of at least two other of the plurality of segments, wherein no portion comprises an entire slice;
generating, by the at least one processor, metadata that represents at least the respective encoding of the plurality of data chunks;
packaging, by the at least one processor, each of the data chunks with at least the metadata;
assigning, by the at least one processor, each of the packages to respective remote storage nodes; and
transmitting, by the at least one processor, each of the packages to the respectively assigned remote storage node.
2. The method of claim 1 , wherein the step of packaging includes erasure coding, wherein the metadata is encoded and not visible to unauthorized users.
3. The method of claim 2 , further comprising abstracting, by the at least one processor, the metadata with two or more of: additional metadata associated with a respective remote storage node; a configuration of a data vault; a hyperlink to an active data vault; and information representing a current state of data blocks.
4. The method of claim 1 , wherein the metadata includes information for reconstructing related segments from corresponding packages and/or information for reconstructing the at least one data file from the plurality of segments.
5. The method of claim 1 , wherein each of the packages include at least some redundant information from at least one other package.
6. The method of claim 1 , further comprising determining by the at least one processor at least one parameter representing at least one of available network bandwidth, geographic proximity, and node availability, wherein selection of respective remote storage nodes is made as a function of the at least one parameter.
7. The method of claim 1 , further comprising applying, by the at least one processor, categories of data, wherein the step of encoding is based at least in part on a respective category.
8. The method of claim 1 , wherein the respective storage nodes are provided as network addressable storage.
9. The method of claim 1 , further comprising providing, by the at least one processor, a graphical user interface that is configured to display at least one map showing locations of the respective storage nodes and a respective operational status of the respective storage nodes.
10. The method of claim 9 , wherein the graphical user interface includes an interactive dashboard that identifies information associated with available storage space, used space, and a number of stored data objects.
11. A system for providing secure distributed storage and transmission of electronic content over at least one communication network, the system comprising:
at least one processor, wherein the at least one processor is configured by executing instructions stored on non-transitory processor-readable media to perform the steps of:
receiving at least one data file;
parsing the at least one data file into a plurality of segments, wherein each one of the segments has a respective size; dividing each of the plurality of segments into a plurality of slices, wherein each one of the slices has a respective size;
encoding a plurality of data chunks, each data chunk comprising i] a portion of at least two of the slices from a first segment and ii) a plurality of portions of at least one respective slice from each of at least two other of the plurality of segments, wherein no portion comprises an entire slice;
generating, by the at least one processor, metadata that represents at least the respective encoding of the plurality of data chunks;
packaging each of the data chunks with at least the metadata;
assigning each of the packages to respective remote storage nodes; and
transmitting each of the packages to the respectively assigned remote storage node.
12. The system of claim 11 , wherein the step of packaging includes erasure coding, wherein the metadata is encoded and not visible to unauthorized users.
13. The system of claim 12 , wherein the non-transitory processor-readable media further have instructions that, when executed by the at least one processor, cause the at least one processor to abstract the metadata with two or more of: additional metadata associated with a respective remote storage node; a configuration of a data vault; a hyperlink to an active data vault; and information representing a current state of data blocks.
14. The system of claim 11 , wherein the metadata includes information for reconstructing related segments from corresponding packages and/or information for reconstructing the at least one data file from the plurality of segments.
15. The system of claim 11 , wherein each of the packages include at least some redundant information from at least one other package.
16. The system of claim 11 , wherein the non-transitory processor-readable media further have instructions that, when executed by the at least one processor, cause the at least one processor to determine at least one parameter representing at least one of available network bandwidth, geographic proximity, and node availability, wherein selection of respective remote storage nodes is made as a function of the at least one parameter.
17. The system of claim 11 , wherein the non-transitory processor-readable media further have instructions that, when executed by the at least one processor, cause the at least one processor to apply categories of data, wherein the step of encoding is based at least in part on a respective category.
18. The system of claim 11 , wherein the respective storage nodes are provided as network addressable storage.
19. The system of claim 11 , wherein the non-transitory processor-readable media further have instructions that, when executed by the at least one processor, cause the at least one processor to provide a graphical user interface that is configured to display at least one map showing locations of the respective storage nodes and a respective operational status of the respective storage nodes.
20. The system of claim 19 , wherein the graphical user interface includes an interactive dashboard that identifies information associated with available storage space, used space, and a number of stored data objects.