Code block resynchronization for distributed multi-mirror erasure coding system

1. A method of resynchronizing a node of a distributed storage system with other nodes of the distributed storage system, comprising:

identifying an out-of-sync block of the node;

determining that the out-of-sync block is a code block, wherein the code block is generated by performing an erasure coding operation on data blocks which are stored in the other nodes, wherein:

determining that the out-of-sync block is a code block is based on determining that the out-of-sync block maps to an address space maintained for code blocks including the out-of-sync block, and

the address space maintained for code blocks is different from an address space maintained for a plurality of data blocks including the data blocks;

locating a mirrored code block in an address space maintained for mirrored code blocks; and

storing contents associated with the mirrored code block in a storage location of the out-of-sync block.

2. The method of claim 1 , further comprising:

determining that the contents of the mirrored code block can be read from a storage location of the mirrored code block;

based on the determining:

reading the contents of the mirrored code block from the storage location of the mirrored code block, wherein the contents associated with the mirrored code block comprise the read contents.

3. The method of claim 1 , further comprising:

determining that the contents of the mirrored code block cannot be read from a storage location of the mirrored code block;

based on the determining:

reconstructing the contents of the mirrored code block from the data blocks which are stored in the other nodes, wherein the contents associated with the mirrored code block comprise the reconstructed contents.

4. The method of claim 1 , wherein

a storage location of the mirrored code block is identified in the address space maintained for mirrored code blocks based on the storage location of the out-of-sync block in the address space maintained for code blocks including the out-of-sync block.

5. The method of claim 4 , further comprising:

maintaining first metadata for the code blocks including the out-of-sync block, the first metadata defining the address space maintained for code blocks including the out-of-sync block; and

maintaining second metadata for the plurality of data blocks including the data blocks which are stored in the other nodes, the second metadata defining the address space maintained for the plurality of data blocks including the data blocks which are stored in the other nodes.

6. The method of claim 1 , wherein identifying the out-of-sync block of the node comprises:

generating a resynching bitmap of blocks of the node, the resynching bitmap indicating which blocks of the node are out-of-sync based on write operations that were carried out in the other nodes while the node was offline.

7. The method of claim 6 , wherein generating the resynching bitmap comprises generating the resynching bitmap using a plurality of tracking bitmaps instantiated to keep track of the write operations carried out in the other nodes while the node was offline.

8. A non-transitory computer readable storage medium comprising instructions that are executable in a computer system to cause the computer system to perform a method of resynchronizing a node of a distributed storage system with other nodes of the distributed storage system, the method comprising:

identifying an out-of-sync block of the node;

determining that the out-of-sync block is a code block, wherein the code block is generated by performing an erasure coding operation on data blocks which are stored in the other nodes, wherein:

determining that the out-of-sync block is a code block is based on determining that the out-of-sync block maps to an address space maintained for code blocks including the out-of-sync block, and

the address space maintained for code blocks is different from an address space maintained for a plurality of data blocks including the data blocks;

locating a mirrored code block in an address space maintained for mirrored code blocks; and

storing contents associated with the mirrored code block in a storage location of the out-of-sync block.

9. The non-transitory computer readable storage medium of claim 8 , wherein the method further comprises:

determining that the contents of the mirrored code block can be read from a storage location of the mirrored code block;

based on the determining:

reading the contents of the mirrored code block from the storage location of the mirrored code block, wherein the contents associated with the mirrored code block comprise the read contents.

10. The non-transitory computer readable storage medium of claim 8 , wherein the method further comprises:

determining that the contents of the mirrored code block cannot be read from a storage location of the mirrored code block;

based on the determining:

reconstructing the contents of the mirrored code block from the data blocks which are stored in the other nodes, wherein the contents associated with the mirrored code block comprise the reconstructed contents.

11. The non-transitory computer readable storage medium of claim 8 , wherein

a storage location of the mirrored code block is identified in the address space maintained for mirrored code blocks based on the storage location of the out-of-sync block in the address space maintained for code blocks including the out-of-sync block.

12. The non-transitory computer readable storage medium of claim 11 , wherein the method further comprises:

maintaining first metadata for the code blocks including the out-of-sync block, the first metadata defining the address space maintained for code blocks including the out-of-sync block; and

maintaining second metadata for the plurality of data blocks including the data blocks which are stored in the other nodes, the second metadata defining the address space maintained for the plurality of data blocks including the data blocks which are stored in the other nodes.

13. The non-transitory computer readable storage medium of claim 8 , wherein identifying the out-of-sync block of the node comprises:

generating a resynching bitmap of blocks of the node, the resynching bitmap indicating which blocks of the node are out-of-sync based on write operations that were carried out in the other nodes while the node was offline.

14. The non-transitory computer readable storage medium of claim 13 , wherein generating the resynching bitmap comprises generating the resynching bitmap using a plurality of tracking bitmaps instantiated to keep track of the write operations carried out in the other nodes while the node was offline.

15. A computer system for resynchronizing a node of a distributed storage system with other nodes of the distributed storage system, comprising:

a memory; and

a processor configured to:

identify an out-of-sync block of the node;

determine that the out-of-sync block is a code block, wherein the code block is generated by performing an erasure coding operation on data blocks which are stored in the other nodes, wherein:

the processor being configured to determine that the out-of-sync block is a code block is based on determining that the out-of-sync block is a code block is based on determining that the out-of-sync block maps to an address space maintained for code blocks including the out-of-sync block, and

the address space maintained for code blocks is different from an address space maintained for a plurality of data blocks including the data blocks;

locate a mirrored code block in an address space maintained for mirrored code blocks; and

store contents associated with the mirrored code block in a storage location of the out-of-sync block.

16. The computer system of claim 15 , wherein the processor is further configured to:

determine that the contents of the mirrored code block can be read from the storage location of the mirrored code block;

based on the determining:

read the contents of the mirrored code block from the storage location of the mirrored code block, wherein the contents associated with the mirrored code block comprise the read contents.

17. The computer system of claim 15 , wherein the processor is further configured to:

determine that the contents of the mirrored code block cannot be read from a storage location of the mirrored code block;

based on the determining:

reconstruct the contents of the mirrored code block from the data blocks which are stored in the other nodes, wherein the contents associated with the mirrored code block comprise the reconstructed contents.

18. The computer system of claim 15 , wherein:

a storage location of the mirrored code block is identified in the address space maintained for mirrored code blocks based on the storage location of the out-of-sync block in the address space maintained for code blocks including the out-of-sync block.

19. The computer system of claim 18 , wherein the processor is further configured to:

maintain first metadata for the code blocks including the out-of-sync block, the first metadata defining the address space maintained for code blocks including the out-of-sync block; and

maintain second metadata for the plurality of data blocks including the data blocks which are stored in the other nodes, the second metadata defining the address space maintained for the plurality of data blocks including the data blocks which are stored in the other nodes.

20. The computer system of claim 15 , wherein the processor being configured to identify the out-of-sync block of the node comprises the processor being configured to:

generate resynching bitmap of blocks of the node, the resynching bitmap indicating which blocks of the node are out-of-sync based on write operations that were carried out in the other nodes while the node was offline.

21. The computer system of claim 20 , wherein the processor being configured to generate the resynching bitmap comprises the processor being configured to generate the resynching bitmap using a plurality of tracking bitmaps instantiated to keep track of the write operations carried out in the other nodes while the node was offline.