Container-based erasure coding
A repository of replicated chunk files is analyzed to identify chunk files that meet at least a portion of combination criteria. Selected chunk files are associated together under a data protection grouping container. Erasure coding is applied to the data protection grouping container including by utilizing the selected chunk files as different data stripes of the erasure coding and generating one or more parity stripes based on the different data stripes.
1 . A method, comprising:
selecting one or more replicated chunk files stored on a first storage device associated with an original chunk file stored on a second storage device different than the first storage device;
updating a metadata table to indicate the one or more replicated chunk files selected are associated with a data protection grouping container without updating metadata within the one or more replicated chunk files associated with the data protection grouping container; and
applying erasure coding to the data protection grouping container, including utilizing the one or more replicated chunk files associated with the data protection grouping container as different data stripes of the erasure coding and generating one or more parity stripes based on the different data stripes.
2 . The method of claim 1 , further comprising:
selecting the one or more replicated chunk files based on combination criteria including at least the one or more replicated chunk files stored on the first storage device different from the second storage device to which the original chunk file is stored.
3 . The method of claim 2 , further comprising:
selecting the one or more replicated chunk files based on the combination criteria further including on one or more of:
an age of one of the one or more replicated chunk files selected, a size of the one of the one or more replicated chunk files selected, whether the one of the one or more replicated chunk files selected includes non-deduplicated data chunks, a storage node that includes the first storage device or the second storage device storing the one of the one or more replicated chunk files selected, a chassis including the storage node that includes the first storage device or the second storage device storing the one of the one or more replicated chunk files selected, and/or a rack including the chassis including the storage node that includes the first storage device or the second storage device storing the one of the one or more replicated chunk files selected.
4 . The method of claim 1 , further comprising:
associating the one or more replicated chunk files selected in the data protection grouping container by updating the metadata table to indicate the one or more replicated chunk files are associated with the data protection grouping container without writing the one or more replicated chunk files associated with the data protection grouping container to a new chunk file.
5 . The method of claim 4 , further comprising:
updating the metadata table to include an entry, corresponding to the data protection grouping container, that associates the one or more replicated chunk files selected with a corresponding storage device of a storage system.
6 . The method of claim 1 , further comprising:
analyzing a repository of replicated chunk files to select the one or more replicated chunk files; and
wherein the analyzing includes selecting and including a first replicated chunk file in the data protection grouping container and selecting a second replicated chunk file based on the second replicated chunk file satisfying combination criteria.
7 . The method of claim 6 , further comprising:
determining whether the second replicated chunk file satisfies the combination criteria.
8 . The method of claim 7 , further comprising:
associating the second replicated chunk file with the data protection grouping container based on determining the second replicated chunk file satisfies the combination criteria.
9 . The method of claim 8 , further comprising determining whether the data protection grouping container is full based on an erasure coding configuration.
10 . The method of claim 9 , wherein the erasure coding configuration indicates a number of data stripes and a number of parity stripes to include in the data protection grouping container.
11 . The method of claim 1 , wherein the one or more replicated chunk files selected and one or more parity stripes generated are stored on different storage devices of a storage system.
12 . The method of claim 1 , further comprising:
monitoring the data protection grouping container; and
performing a garbage collection process.
13 . The method of claim 12 , wherein the garbage collection process includes determining whether a measure of unreferenced data chunks associated with the data protection grouping container is greater than a threshold measure of unreferenced data chunks.
14 . The method of claim 13 , further comprising:
generating a new data protection grouping container based on whether the measure of unreferenced data chunks associated with the data protection grouping container is greater than the threshold measure of unreferenced data chunks.
15 . The method of claim 14 , further comprising:
updating the data protection grouping container based on whether the measure of unreferenced data chunks associated with the data protection grouping container is greater than the threshold measure of unreferenced data chunks.
16 . Non-transitory computer-readable storage media encoded with instructions that, when executed, cause one or more processors to:
select one or more replicated chunk files stored on a first storage device associated with an original chunk file stored on a second storage device different than the first storage device;
update a metadata table to indicate the one or more replicated chunk files selected are associated with a data protection grouping container without updating metadata within the one or more replicated chunk files associated with the data protection grouping container; and
apply erasure coding to the data protection grouping container, including utilizing the one or more replicated chunk files associated with the data protection grouping container as different data stripes of the erasure coding and generating one or more parity stripes based on the different data stripes.
17 . The non-transitory computer-readable storage media of claim 16 , wherein the instructions, when executed, further cause the one or more processors to:
select the one or more replicated chunk files based on combination criteria including at least the one or more replicated chunk files stored on the first storage device different from the second storage device to which the original chunk file is stored.
18 . The non-transitory computer-readable storage media of claim 16 , wherein the instructions, when executed, further cause the one or more processors to:
associate the one or more replicated chunk files selected in the data protection grouping container by updating the metadata table to indicate the one or more replicated chunk files are associated with the data protection grouping container without writing the one or more replicated chunk files associated with the data protection grouping container to a new chunk file.
19 . A system, comprising:
one or more processors; and
memory, coupled to the one or more processors, the memory storing instructions that when executed cause the one or more processors to:
select one or more replicated chunk files stored on a first storage device associated with an original chunk file stored on a second storage device different than the first storage device;
update a metadata table to indicate the one or more replicated chunk files selected are associated with a data protection grouping container without updating metadata within the one or more replicated chunk files associated with the data protection grouping container; and
apply erasure coding to the data protection grouping container, including utilizing the one or more replicated chunk files associated with the data protection grouping container as different data stripes of the erasure coding and generating one or more parity stripes based on the different data stripes.
20 . The system of claim 19 , wherein the instructions, when executed, further cause the one or more processors to:
associate the one or more replicated chunk files selected in the data protection grouping container by updating the metadata table to indicate the one or more replicated chunk files are associated with the data protection grouping container without writing the one or more replicated chunk files associated with the data protection grouping container to a new chunk file.