Disaster recovery with distributed erasure coding
Processing load for remote zones and inter-zone network traffic during geographically distributed (GEO) erasure coding can be reduced. In one aspect, GEO erasure coding can be performed in two phases, wherein each phase can be distributed between zones. Moreover, during a first phase, partially-finished coding chunks can be generated and provided to the remote zones. Further, during a second phase the remote zones can combine the received partially-finished coding chunks independently, to generate a complete coding chunk that can be utilized for data recovery.
1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
receiving, by a back-end storage zone of the system, partial coding chunks from respective front-end storage zones of a geographically distributed storage system, wherein each partial coding chunk is determined based on performing a distinct portion of a first phase of an erasure coding operation that encodes a distinct data chunk of distinct data chunks of a data element, where each distinct data chunk is stored within a different front-end storage zone of the respective front-end storage zones; and
performing, by the back-end storage zone, a second phase of the erasure coding operation that comprises combining the partial coding chunks to encode a final coding chunk.
2. The system of claim 1 , wherein the combining comprises summing the partial coding chunks.
3. The system of claim 1 , wherein each partial coding chunk is determined based on the distinct portion of a defined coding matrix data of the erasure coding operation.
4. The system of claim 3 , wherein the defined coding matrix data comprises a set of defined coefficients, and wherein a first partial coding chunk of the partial coding chunks is determined based on a result of multiplying a first distinct data chunk of the distinct data chunks with a coefficient of the set of defined coefficients.
5. The system of claim 1 , wherein the operations further comprise:
subsequent to encoding of the final coding chunk, deleting, by the back-end storage zone, the partial coding chunks.
6. The system of claim 1 , wherein the final coding chunk is employable to recover at least one of the distinct data chunks during a failure.
7. The system of claim 6 , wherein the least one of the distinct data chunks is recovered based on defined decoding matrix data.
8. The system of claim 1 , wherein the geographically distributed storage system comprises a geographically distributed object storage system.
9. The system of claim 1 , wherein the operations further comprise:
storing, by the back-end storage zone, the final coding chunk within a data store of another back-end storage zone of the geographically distributed storage system.
10. A method, comprising:
receiving, by a back-end storage zone of a system comprising a processor, partial coding chunks from respective front-end storage zones of a geographically distributed storage system, wherein the partial coding chunks are respectively determined based on partially encoding a respective distinct data chunk of distinct data chunks of a data item by utilizing a distinct portion of a first phase of a defined erasure coding scheme, where the distinct data chunks are stored within a different front-end storage zone of the respective front-end storage zones; and
performing, by the back-end storage zone of the system, a second phase of the erasure coding scheme that comprises aggregating the partial coding chunks to encode a final coding chunk that is employable for recovery of at least one of the distinct data chunks during a failure.
11. The method of claim 10 , further comprising:
storing, by the back-end storage zone of the system, the final coding chunk within a data store of another back-end storage zone of the geographically distributed storage system.
12. The method of claim 11 , further comprising:
in response to the storing, deleting, by the back-end storage zone of the system, the partial coding chunks.
13. The method of claim 10 , wherein the partial coding chunks have been determined based on a respective distinct portion of coding matrix data associated with the defined erasure coding scheme.
14. The method of claim 13 , wherein the coding matrix data comprises a set of defined coefficients, and wherein a first partial coding chunk of the partial coding chunks has been determined based on a combination of a first distinct data chunk of the distinct data chunks with a coefficient of the set of defined coefficients.
15. A non-transitory computer-readable medium comprising instructions that, in response to execution, cause a server device comprising a processor to perform operations, comprising:
in response to determining that distinct data chunks of a data element stored within respective front-end zones of a geographically distributed storage system are to be protected via erasure coding, determining groups of partial coding chunks, wherein each group of the groups of partial coding chunks comprises different partially encoded versions of the distinct data chunks generated by the respective front-end zones by performing a distinct portion of a first phase of an erasure coding operation; and
facilitating a transfer of the groups of partial coding chunks to respective back-end zones of the geographically distributed storage system to facilitate encoding, by a second phase of the erasure coding operation, respective complete coding chunks that are employable to recover at least one of the distinct data chunks during a failure.
16. The non-transitory computer-readable medium of claim 15 , wherein the determining comprises determining the groups of partial coding chunks based on defined coding matrix data.
17. The non-transitory computer-readable medium of claim 16 , wherein the defined coding matrix data is indicative of coefficients.
18. The non-transitory computer-readable medium of claim 17 , wherein the groups of partial coding chunks are determined based on combining the distinct data chunks with different corresponding coefficients of the coefficients.
19. The non-transitory computer-readable medium of claim 15 , wherein a complete coding chunk of the complete coding chunks is encoded based on the second phase of the erasure coding operation that comprises a summation of the different partially encoded versions of the distinct data chunks of a group of partial coding chunks.
20. The non-transitory computer-readable medium of claim 15 , wherein the geographically distributed storage system comprises a geographically distributed object storage system.