System and method of reliable distributed data storage with controlled redundancy

1. A method for data storage in a distributed data storage system, the method comprising:

generating a plurality of derivatives from a file using an (n,k) algorithm, wherein the (n,k) algorithm provides that a minimum number k of the derivatives is required to restore the file;

storing the derivatives in a plurality of nodes in a multilevel tree, wherein the multilevel tree includes at least L levels and the plurality of nodes are geographically distributed data storage modules;

determining a rank r l,m indicating a fault tolerance of the file, wherein the rank r l,m represents a number of derivatives available after a failure on at least one level l of the L levels with a number of failed nodes m in the level l;

determining whether the rank r l,m is greater than or equal to the minimum number k;

responsive to determining that the rank r l,m is less than the minimum number k, storing at least one of the plurality of derivatives in a different node of the plurality of nodes or storing at least one additional derivative from the file in one of the plurality of nodes.

2. The method of claim 1 , further comprising verifying an acceptable fault tolerance of the file if the rank r l,m ≥k.

3. The method of claim 1 , further comprising reassembling the file from the distributed storage system using the (n,k) algorithm.

4. The method of claim 1 , wherein at least one of k≥3 and m≥2.

5. The method of claim 1 , further comprising:

generating a plurality of derivatives from at least one additional file using the (n,k) algorithm;

storing the derivatives from the at least one additional file in the multilevel tree, such that the rank r l,m of the at least one additional file is greater than or equal to k,

wherein if a number of nodes on one level l of the L levels of the multilevel tree is equal to q, then for the at least one additional file, a relation of q≥m+1 is satisfied, and a minimum number of the nodes on the one level l satisfies q min =m+1.

6. The method of claim 1 , further comprising storing the derivatives in the plurality of nodes in the multilevel tree, such that none of the derivatives are duplicates.

7. The method of claim 1 , further comprising determining the rank r l,m indicating the fault tolerance of the file, such that the rank r l,m is equal to a sum p l,m of the number p of derivatives over all the elements of the level l, except for m with a biggest values of p.

8. The method of claim 1 , wherein the storing of the derivatives in the plurality of nodes comprises:

storing a number of k derivatives on q−m nodes for one level l of the levels L, where q−m is a number of available nodes on the level l; and

storing remaining derivatives on the q-m nodes for the one level l,

wherein a total number of derivatives will be k+(round ((k−1)/(q−m)) +1) * m.

9. The method of claim 1 , wherein the determining of the rank r l,m indicating of the file comprises calculating table of a file reliability matrix having a plurality of cells that each indicate a rank for each level l of the levels L and a corresponding number of node failures m of the level l.

10. A system for data storage, the system comprising:

a distributed data storage system including a plurality of nodes arranged in a multilevel tree having at least L levels, wherein the plurality of nodes are geographically distributed data storage modules; and

a processor configured to:

generate a plurality of derivatives from a file using an (n,k) algorithm, wherein the (n,k) algorithm provides that a minimum number k of the derivatives is required to restore the file,

store the derivatives in the plurality of nodes in the plurality of nodes,

determine a rank r l,m indicating a fault tolerance of the file, wherein the rank r l,m represents a number of derivatives available after a failure on at least one level l of the L levels with a number of failed nodes m in the level l,

determine whether the rank r l,m is greater than or equal to the minimum number k,

responsive to determining that the rank r l,m is less than the minimum number k, store at least one of the plurality of derivatives in a different node of the plurality of nodes or store at least one additional derivative from the file in one of the plurality of nodes.

11. The system of claim 10 , wherein the processor is further configured to verify an acceptable fault tolerance of the file if the rank r l,m ≥k.

12. The system of claim 10 , wherein the processor is further configured to reassemble the file from the distributed storage system using the (n,k) algorithm.

13. The system of claim 10 , wherein at least one of k≥3 and m≥2.

14. The system of claim 10 , wherein the processor is further configured to:

store a number of k derivatives on q−m nodes for one level l of the levels L, where q−m is a number of available nodes on the level l; and

store remaining derivatives on the q−m nodes for the one level l,

wherein a total number of derivatives will be k+(round ((k−1)/(q−m)) +1) * m.

15. The system of claim 10 , wherein the processor is further configured to evaluate the rank r l,m indicating of the file by calculating table of a file reliability matrix having a plurality of cells that each indicate a rank for each level l of the levels L and a corresponding number of node failures m of the level l, such that the file reliability matrix indicates a fault tolerance state of the file in the distributed data storage system.

16. The system of claim 10 , wherein if a top level of the multilevel tree of the distributed data storage system fails, then all nodes on lower levels of the multilevel tree become unavailable.

17. A non-transitory computer readable medium storing computer executable instructions for storing data in a distributed data storage system, including instructions for:

generating a plurality of derivatives from a file using an (n,k) algorithm, wherein the (n,k) algorithm provides that a minimum number k of the derivatives is required to restore the file;

storing the derivatives in a plurality of nodes in a multilevel tree, wherein the multilevel tree includes at least L levels and the plurality of nodes are geographically distributed data storage modules;

determining a rank r l,m indicating a fault tolerance of the file, wherein the rank r l,m represents a number of derivatives available after a failure on at least one level l of the L levels with a number of failed nodes m in the level l;

determining whether the rank r l,m is greater than or equal to the minimum number k;

responsive to determining that the rank r l,m is less than the minimum number k, storing at least one of the plurality of derivatives in a different node of the plurality of nodes or storing at least one additional derivative from the file in one of the plurality of nodes.

18. The non-transitory computer readable medium according to claim 17 , further including instructions for:

generating a plurality of derivatives from at least one additional file using the (n,k) algorithm;

storing the derivatives from the at least one additional file in the multilevel tree, such that the rank r l,m of the at least one additional file is greater than or equal to k,

wherein if a number of nodes on one level l of the L levels of the multilevel tree is equal to q, then for the at least one additional file, a relation of q≥m+1is satisfied, and a minimum number of the nodes on the one level l satisfies q min =m+1.

19. The non-transitory computer readable medium according to claim 17 , further including instructions for determining the rank R l,m indicating the fault tolerance of the file, such that the rank r l,m is equal to a sum p l,m of the number p of derivatives over all the elements of the level l, except form with a biggest values of p.