Performing an in-line erasure coding process using a write-ahead log
A plurality of data stripes and one or more parity stripes are generated using a plurality of data chunks stored in a write-ahead log based on an erasure coding configuration. The plurality of data stripes and the one or more parity stripes are stored on corresponding different storage devices. The plurality of data stripes and the one or more parity stripes are associated together under a data protection grouping container.
1. A method, comprising:
generating, based on an erasure coding configuration, a plurality of chunk file groups from a plurality of data chunks stored in a write-ahead log;
generating, based on the plurality of chunk file groups, a plurality of data stripes and one or more parity stripes using the plurality of data chunks stored in the write-ahead log;
storing the plurality of data stripes and the one or more parity stripes on corresponding different storage devices; and
associating together the plurality of data stripes and the one or more parity stripes under a data protection grouping container.
2. The method of claim 1 , further comprising removing the plurality of the data chunks from the write-ahead log.
3. The method of claim 1 , wherein each of the plurality of data stripes is a corresponding chunk file.
4. The method of claim 1 , further comprising ingesting source data from a source system.
5. The method of claim 4 , wherein ingesting the source data from the source system includes chunking the data into the plurality of data chunks or receiving the plurality of data chunks from the source system.
6. The method of claim 1 , wherein generating, based on the plurality of chunk file groups, the plurality of data stripes and the one or more parity stripes using the plurality of data chunks stored in the write-ahead log includes deduplicating the plurality of data chunks stored in the write-ahead log.
7. The method of claim 6 , further comprising determining whether a threshold size of data chunks remain in the write-ahead log.
8. The method of claim 7 , wherein the erasure coding configuration is maintained based on determining the threshold size of data chunks remain in the write-ahead log.
9. The method of claim 7 , further comprising waiting a threshold period of time based on a determination that the threshold size of data chunks do not remain in the write-ahead log.
10. The method of claim 7 , further comprising selecting an alternate erasure coding configuration based on a determination that the threshold size of data chunks do not remain in the write-ahead log.
11. The method of claim 10 , wherein the alternate erasure coding configuration decreases a number of data stripes included in the erasure coding configuration.
12. The method of claim 10 , wherein the alternate erasure coding configuration decreases a size of a data stripe included in the erasure coding configuration.
13. The method of claim 1 , further comprising updating metadata that enables the plurality of data chunks to be located.
14. The method of claim 13 , wherein updating the metadata that enables the plurality of data chunks to be located includes selecting a data chunk included in one of the plurality of data stripes.
15. The method of claim 14 , wherein updating the metadata that enables the plurality of data chunks to be located further includes determining whether an entry of a chunk metadata data structure corresponding to the data chunk has been updated.
16. The method of claim 15 , wherein updating the metadata that enables the plurality of data chunks to be located further includes updating an entry of a chunk file metadata data structure not to reference the data chunk based on a determination that the entry of the chunk metadata data structure corresponding to the data chunk has been updated.
17. The method of claim 15 , wherein updating the metadata that enables the plurality of data chunks to be located further includes updating the entry of the chunk metadata data structure to reference a chunk file storing the data chunk.
18. A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
generating, based on an erasure coding configuration, a plurality of chunk file groups from a plurality of data chunks stored in a write-ahead log;
generating, based on the plurality of chunk file groups, a plurality of data stripes and one or more parity stripes using the plurality of data chunks stored in the write-ahead log;
storing the plurality of data stripes and the one or more parity stripes on corresponding different storage devices; and
associating together the plurality of data stripes and the one or more parity stripes under a data protection grouping container.
19. A system, comprising:
a processor configured to:
generate, based on an erasure coding configuration, a plurality of chunk file groups from a plurality of data chunks stored in a write-ahead log;
generate, based on the plurality of chunk file groups, a plurality of data stripes and one or more parity stripes using the plurality of data chunks stored in the write-ahead log;
store the plurality of data stripes and the one or more parity stripes on corresponding different storage devices; and
associate together the plurality of data stripes and the one or more parity stripes under a data protection grouping container; and
a memory coupled to the processor and configured to provide the processor with instructions.