Scale out data protection with erasure coding
Scale out data protection with erasure coding is presented herein. Based on an initial number of storage devices determined to have been included in an initial stage of a data storage cluster, an initial protection scheme for the initial stage can determine first coding fragment(s) for data stored within the data storage cluster to facilitate a first recovery, from the initial stage, of the data using the first coding fragment(s). Further, in response to a defined number of additional storage devices being determined to have been added to the data storage cluster to generate a modified data storage cluster, the initial protection scheme can be modified to obtain a modified protection scheme that can determine, for the modified data storage cluster, second coding fragment(s) for the data to facilitate a second recovery of the data using the first coding fragment(s) and the second coding fragment(s).
1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
based on an initial number of storage devices that have been determined to have been included in an initial stage of a data storage cluster, determining an initial protection scheme for the initial stage of the data storage cluster, wherein the initial protection scheme comprises determining a first group of coding fragments for data that has been stored within the data storage cluster to facilitate a first recovery, from the initial stage of the data storage cluster, of the data using the first group of coding fragments; and
in response to a defined number of additional storage devices being determined to have been added to the data storage cluster to generate a modified data storage cluster, modifying the initial protection scheme to obtain a modified protection scheme for the modified data storage cluster, wherein the modified protection scheme comprises determining a second group of coding fragments, different than the first group of coding fragments, for the data to facilitate a second recovery, from the modified data storage cluster, of the data using the first group of coding fragments and the second group of coding fragments.
2. The system of claim 1 , wherein a sum of a first amount of coding fragments of the first group of coding fragments and a second amount of coding fragments of the second group of coding fragments is less than or equal to a defined upper limit for a number of coding fragments for the data.
3. The system of claim 1 , wherein the initial protection scheme comprises:
storing the first group of coding fragments in the data storage cluster to obtain a first stored portion of coding fragments.
4. The system of claim 3 , wherein the modified protection scheme comprises:
storing the second group of coding fragments in the data storage cluster to obtain a second stored portion of coding fragments.
5. The system of claim 4 , wherein the operations further comprise:
recovering the data from the modified data storage cluster using the first stored portion of coding fragments and the second stored portion of coding fragments.
6. The system of claim 1 , wherein the initial protection scheme and the modified protection scheme comprise an erasure coding-based protection scheme.
7. The system of claim 6 , wherein a target coding matrix that is based on a defined upper limit for a number of coding fragments for the data and a defined amount of data fragments representing the data comprises a first sub-matrix corresponding to the initial stage of the data storage cluster and a second sub-matrix corresponding to the modified data storage cluster.
8. The system of claim 7 , wherein the second sub-matrix comprises the first sub-matrix.
9. The system of claim 8 , wherein the first sub-matrix is based on the defined amount of data fragments and a first amount of coding fragments of the first group of coding fragments, wherein the second sub-matrix is based on the defined amount of data fragments and a second amount of coding fragments of the second group of coding fragments, and wherein a sum of the first amount of coding fragments and the second amount of coding fragments is less than or equal to the defined upper limit for the number of coding fragments for the data.
10. A method, comprising:
determining, by a system comprising a processor, a target protection scheme for a data storage cluster, wherein the target protection scheme corresponds to a target coding matrix for data that has been stored within the data storage cluster;
based on an initial number of storage devices that have been included in an initial stage of the data storage cluster, determining, by the system, an initial protection scheme for the initial stage of the data storage cluster comprising determining a first portion of coefficients of the target coding matrix to facilitate a first recovery, from the initial stage of the data storage cluster, of a lost portion of the data; and
in response to determining that a defined number of additional storage devices have been added to the data storage cluster to generate a modified data storage cluster, modifying, by the system, the initial protection scheme to obtain a modified protection scheme for the modified data storage cluster, wherein the modified protection scheme comprises determining a second portion of coefficients of the target coding matrix that is different than the first portion of coefficients, and wherein the first portion of coefficients and the second portion of coefficients facilitate a second recovery, from the modified data storage cluster, of the lost portion of the data.
11. The method of claim 10 , wherein the operations further comprise:
recovering, by the system, the lost portion of the data from the modified data storage cluster using the first portion of coefficients and the second portion of coefficients.
12. The method of claim 10 , wherein the initial protection scheme and the modified protection scheme comprise an erasure coding-based protection scheme.
13. The method of claim 10 , further comprising
determining, by the system, the target coding matrix based on a defined upper limit for a number of coding fragments for the data and a defined number of data fragments representing the data.
14. The method of claim 13 , wherein the determining the first portion of coefficients comprises determining the first portion of coefficients based on the defined number of data fragments representing the data.
15. The method of claim 14 , wherein the determining the second portion of coefficients comprises determining the second portion of coefficients based on the defined number of data fragments representing the data.
16. A non-transitory machine-readable medium comprising instructions that, in response to execution, cause a system comprising a processor to perform operations, comprising:
determining a target protection scheme for a data storage cluster, wherein the target protection scheme corresponds to a defined upper limit of coding fragments for data that has been stored within the data storage cluster;
based on an initial number of storage devices that have been determined to have been included in an initial stage of the data storage cluster, determining an initial protection scheme for the initial stage of the data storage cluster comprising determining a first portion of the coding fragments to facilitate a first recovery, from the initial stage of the data storage cluster, of the data; and
in response to a defined number of additional storage devices being determined to have been added to the data storage cluster to obtain a modified data storage cluster, modifying the initial protection scheme to obtain a modified protection scheme for the modified data storage cluster, wherein the modified protection scheme comprises determining a second portion of the coding fragments that is different than the first portion of the coding fragments, and wherein the first portion of the coding fragments and the second portion of the coding fragments facilitate a second recovery, from the modified data storage cluster, of the data.
17. The non-transitory machine-readable medium of claim 16 , wherein the target protection scheme, the initial protection scheme, and the modified protection scheme comprise an erasure coding-based protection scheme.
18. The non-transitory machine-readable medium of claim 17 , wherein the operations further comprise:
storing the first portion of the coding fragments in the data storage cluster to obtain a first stored portion of the coding fragments; and
storing the second portion of the coding fragments in the data storage cluster to obtain a second stored portion of the coding fragments.
19. The non-transitory machine-readable medium of claim 18 , wherein the operations further comprise:
recovering the data from the modified data storage cluster using the first stored portion of the coding fragments and the second stored portion of the coding fragments.
20. The non-transitory machine-readable medium of claim 19 , wherein the first stored portion of the coding fragments and the second stored portion of the coding fragments correspond to a defined amount of data fragments representing the data.