Distributed erasure coded virtual file system

1. A system comprising:

a computing device comprising a plurality of storage devices, wherein:

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans two or more of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

said computing device is operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains;

said plurality of FEC protection domains comprises a plurality of first storage blocks and a second storage block;

stored in each of said plurality of first storage blocks is either: a first data portion of a plurality of data portions, or protection bits calculated based on said plurality of data portions; and

stored in said second storage block is a protection portion calculated based on contents of said plurality of first storage blocks.

2. The system of claim 1 , wherein said rank is based on how many storage blocks are in each of said plurality of stripes.

3. The system of claim 2 , wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

4. The system of claim 1 , wherein said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

5. The system of claim 1 , wherein said rank is based on read and write overhead for committing to each of said plurality of stripes.

6. The system of claim 1 , wherein said plurality of storage devices are organized into a plurality of failure domains.

7. The system of claim 6 , wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

8. The system of claim 6 , wherein each one of said plurality of stripes spans said plurality of failure domains.

9. The system of claim 6 , wherein, in an instance when multiple of said plurality of stripes have one or more failed blocks, said computing device is operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

10. The system of claim 9 , wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said computing device performs said reconstruction of said plurality of stripes which have only one failed block.

11. The system of claim 1 , wherein said computing device is operable to:

perform successive commit operations to a selected one of said plurality of stripes until said selected one of said plurality of stripes no longer meets a determined criterion; and

upon said selected one of said plurality of stripes no longer meeting said determined criterion, select a different one of said plurality of stripes.

12. The system of claim 11 , wherein said criterion is based on how many blocks are available for new data to be written to.

13. The system of claim 1 , wherein said computing device is operable to buffer a second data portion of the plurality of data portions, said second data portion being from a client application write operation unrelated to said first data portion, wherein said computing device is operable to commit said first data portion and said second data portion to a single one of said plurality of stripes.

14. A method for distributing error correction, wherein the method comprises:

distributing a plurality of failure resilient address spaces across a plurality of storage devices such that each of said plurality of failure resilient address spaces spans two or more of said storage devices;

organizing each one of said plurality of failure resilient address spaces into a plurality of stripes;

organizing each one or more stripes of said plurality of stripes into a respective one of a plurality of forward error correction (FEC) protection domains, each of said plurality of FEC protection domains comprising a plurality of first storage blocks and a second storage block;

storing data in a selected one of a plurality of first storage blocks of an FEC protection domain of said plurality of FEC protection domains;

calculating a protection portion based on contents of said plurality of first storage blocks of said FEC protection domain;

storing said protection portion in a second storage block of said FEC protection domain ranking said plurality of stripes; and

selecting, according to said ranking, which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces.

15. The method of claim 14 , wherein the method comprises storing protection bits calculated based on said data, the protection bits being stored in a different one of said plurality of first storage blocks of said FEC protection domain.

16. A method for distributing error correction, wherein the method comprises:

distributing a plurality of failure resilient address spaces across a plurality of storage devices such that each of said plurality of failure resilient address spaces spans two or more of said storage devices;

organizing each one of said plurality of failure resilient address spaces into a plurality of stripes;

organizing each one or more stripes of said plurality of stripes into a respective one of a plurality of forward error correction (FEC) protection domains, each of said plurality of FEC protection domains comprising a plurality of first storage blocks and a second storage block;

storing data in a selected one of a plurality of first storage blocks of an FEC protection domain of said plurality of FEC protection domains;

calculating a protection portion based on contents of said plurality of first storage blocks of said FEC protection domain;

storing said protection portion in a second storage block of said FEC protection domain; and

prioritizing reconstruction of multiple of said plurality of stripes in descending order according to a number of failed blocks in each of said multiple of said plurality of stripes.

17. The method of claim 14 , wherein the method comprises organizing said plurality of storage devices into a plurality of failure domains; and upon failure of any particular one of said plurality of failure domains, distributing a workload for reconstructing the lost data among each of the others of said plurality of failure domains.

18. The method of claim 14 , wherein the method comprises prioritizing reconstruction of multiple of said plurality of stripes in descending order according to a number of failed blocks in each of said multiple of said plurality of stripes.

19. The method of claim 14 , wherein the method comprises buffering data in a different one of said plurality of first storage blocks, said different one of said plurality of first storage blocks being from a client application write operation unrelated to said selected one of said plurality of first storage blocks; and committing said selected one and said different one of said plurality of first storage blocks to a single one of said plurality of stripes.

20. A system comprising:

a computing device comprising a plurality of storage devices, wherein:

a plurality of failure resilient address spaces are distributed across said plurality of storage devices such that each of said plurality of failure resilient address spaces spans two or more of said storage devices;

each one of said plurality of failure resilient address spaces is organized into a plurality of stripes;

each one or more stripes of said plurality of stripes is part of a respective one of a plurality of forward error correction (FEC) protection domains;

said plurality of FEC protection domains comprises a plurality of first storage blocks and a second storage block;

stored in each of said plurality of first storage blocks is either: a first data portion of a plurality of data portions, or protection bits calculated based on said plurality of data portions;

stored in said second storage block is a protection portion calculated based on contents of said plurality of first storage blocks;

said plurality of storage devices are organized into a plurality of failure domains; and

in an instance when multiple of said plurality of stripes have one or more failed blocks, said computing device is operable to prioritize reconstruction of said multiple of said plurality of stripes in descending order of number of failed blocks in each of said multiple of said plurality of stripes.

21. The system of claim 20 , wherein said computing device is operable to rank said plurality of stripes, wherein said rank is used for selection of which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces.

22. The system of claim 21 , wherein said rank is based on how many storage blocks are in each of said plurality of stripes.

23. The system of claim 22 , wherein, for any particular one of said plurality of stripes, said rank is based on a bitmap stored on said plurality of storage devices with said particular one of said plurality of stripes.

24. The system of claim 21 , wherein said rank is based on how many blocks currently storing data are in each of the plurality of stripes.

25. The system of claim 21 , wherein said rank is based on read and write overhead for committing to each of said plurality of stripes.

26. The system of claim 20 , wherein each of said failure resilient address spaces spans all of said plurality of failure domains, such that upon failure of any particular one of said plurality of failure domains, a workload for reconstructing the lost data is distributed among each of the others of said plurality of failure domains.

27. The system of claim 20 , wherein each one of said plurality of stripes spans said plurality of failure domains.

28. The system of claim 20 , wherein said plurality of computing devices are operable to perform said reconstruction of said plurality of stripes which have two or more failed blocks at a higher rate than a rate at which said computing device performs said reconstruction of said plurality of stripes which have only one failed block.

29. The system of claim 20 , wherein said computing device is operable to:

perform successive commit operations to a selected one of said plurality of stripes until said selected one of said plurality of stripes no longer meets a determined criterion; and

upon said selected one of said plurality of stripes no longer meeting said determined criterion, select a different one of said plurality of stripes.

30. The system of claim 29 , wherein said criterion is based on how many blocks are available for new data to be written to.

31. The system of claim 20 , wherein said computing device is operable to buffer a second data portion of the plurality of data portions, said second data portion being from a client application write operation unrelated to said first data portion, wherein said computing device is operable to commit said first data portion and said second data portion to a single one of said plurality of stripes.

32. The method of claim 16 , wherein the method comprises ranking said plurality of stripes; and selecting, according to said ranking, which of said plurality of stripes to use for a next commit to said particular one of said plurality of failure resilient address spaces.

33. The method of claim 16 , wherein the method comprises storing protection bits calculated based on said data, the protection bits being stored in a different one of said plurality of first storage blocks of said FEC protection domain.

34. The method of claim 16 , wherein the method comprises organizing said plurality of storage devices into a plurality of failure domains; and upon failure of any particular one of said plurality of failure domains, distributing a workload for reconstructing the lost data among each of the others of said plurality of failure domains.

35. The method of claim 16 , wherein the method comprises buffering data in a different one of said plurality of first storage blocks, said different one of said plurality of first storage blocks being from a client application write operation unrelated to said selected one of said plurality of first storage blocks; and committing said selected one and said different one of said plurality of first storage blocks to a single one of said plurality of stripes.