Storing erasure coded data based on reliability of storage devices
The disclosed technology is generally directed towards selecting storage devices, based on predicted reliability, for storing erasure coded data fragments and coding fragments. In general, to increase data availability, data fragments, are stored to more reliable storage devices, while coding fragments are stored to less reliable storage devices. For example, solid state drives (SSDs) tend to fail based on the total number of writes they receive over time, whereby the total number of writes can be used to determine predicted reliability data for an SSD. Before writing the data and coding fragments to a number of storage devices, the storage devices can be sorted based on their predicted reliability such that the data fragments are written to (likely) more reliable devices and coding fragments to less likely storage devices.
1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
obtaining reliability data representative of reliabilities of storage devices;
dividing, based on the reliability data, the storage devices into a higher reliability first group of storage devices according to first reliabilities of the reliabilities of storage devices, a medium reliability second group of storage devices according to second reliabilities of the reliabilities of storage devices, and a lower reliability third group of storage devices according to third reliabilities of the reliabilities of storage devices;
storing, based on the reliability data, erasure coded data fragments to the higher reliability first group of storage devices with the first reliabilities of the reliabilities of storage devices that are more reliable than the medium reliability second group of storage devices with the second reliabilities of the reliabilities of storage devices, wherein the storing the erasure coded data fragments to the higher reliability first group of storage devices comprises storing the erasure coded data fragments to the higher reliability first group of storage devices and to a first part of the medium reliability second group of storage devices; and
storing, based on the reliability data, erasure coded coding fragments to the medium reliability second group of storage devices that are less reliable than the higher reliability first group of storage devices, wherein the storing the erasure coded coding fragments to the medium reliability second group of storage devices comprises storing the erasure coded data fragments to a second part of the medium reliability second group of storage devices and to the lower reliability third group of storage devices.
2. The system of claim 1 , wherein the determining the reliability data comprises determining values representing respective numbers of data writes to the storage devices.
3. The system of claim 2 , wherein the storage devices comprise solid state drives.
4. The system of claim 2 , wherein the storage devices comprise a redundant array of solid state drives.
5. The system of claim 1 , wherein the determining the reliability data comprises determining respective ages of the storage devices.
6. The system of claim 5 , wherein the storage devices comprise hard disk drives.
7. The system of claim 1 , wherein the operations further comprise, erasure coding a chunk data structure into the erasure coded data fragments and the erasure coded coding fragments.
8. The system of claim 1 , wherein the operations further comprise, selecting the storage devices as a group of reliable storage devices, and sorting the group of reliable storage devices based on the reliability data.
9. The system of claim 1 , wherein the operations further comprise, dividing the storage devices into subgroups based on the reliability data, and selecting one or more first subgroups of the subgroups that are more reliable for storing the erasure coded data fragments according to the first reliabilities of the reliabilities of storage devices, and selecting one or more second subgroups of the subgroups that are less reliable for storing the erasure coded coding fragments according to the second reliabilities of the reliabilities of storage devices.
10. The system of claim 1 , wherein the operations further comprise pre-allocating a distributed chunk data structure on the storage devices, comprising allocating data fragment space of the distributed chunk data structure on the higher reliability first group of storage devices, and allocating coding fragment space of the distributed chunk data structure on the medium reliability second group of storage devices.
11. A method, comprising,
obtaining, by a system comprising a processor, reliability data for a group of storage devices;
erasure coding data corresponding to a chunk data structure into data fragments and coding fragments;
dividing, based on the reliability data, the storage devices of the group of storage devices into a higher reliability first subgroup of the group according to first reliabilities of reliabilities of the storage devices, a medium reliability second subgroup of the group according to second reliabilities of the reliabilities of the storage devices, and a lower reliability third subgroup of the group according to third reliabilities of the reliabilities of the storage devices;
storing the data fragments, based on the reliability data, to the first subgroup of the group comprising first storage devices of the group of storage devices that are more reliable than the second subgroup of the group comprising second storage devices of the group of storage devices, wherein the storing the data fragments to the higher reliability first subgroup comprises storing the data fragments to the higher reliability first subgroup and to a first part of the medium reliability second subgroup; and
storing the coding fragments, based on the reliability data, to the medium reliability second subgroup, wherein the storing the coding fragments to the medium reliability second subgroup comprises storing the data fragments to a second part of the medium reliability second subgroup and to the lower reliability third subgroup.
12. The method of claim 11 , wherein the group of storage devices comprise solid state devices, and wherein the obtaining the reliability data for the group of storage devices comprises tracking respective data writes to respective solid state devices.
13. The method of claim 11 , wherein the group of storage devices comprise disk drives, and wherein the obtaining the reliability data for the group of storage devices comprises obtaining respective ages of the disk drives.
14. The method of claim 11 , further comprising sorting the group of storage devices based on the reliability data to determine the higher reliability first subgroup and the medium reliability second subgroup.
15. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of a data storage system, facilitate performance of operations, the operations comprising:
obtaining reliability data representative of respective reliabilities associated with a group of storage devices;
determining, based on the reliability data, a higher reliability first subgroup of the group according to first reliabilities of the respective reliabilities, a medium reliability second subgroup of the group according to second reliabilities of the respective reliabilities, and a lower reliability third subgroup of the group according to third reliabilities of the respective reliabilities;
allocating data fragments of a distributed chunk data structure on the higher reliability first subgroup, wherein the allocating the coding fragments of the distributed chunk to the higher reliability first subgroup comprises allocating the data fragments of the distributed chunk data structure to the higher reliability first subgroup and to a first part of the medium reliability second subgroup; and
allocating coding fragments of the distributed chunk data structure on the medium reliability second subgroup, wherein the allocating the coding fragments of the distributed chunk data structure to the medium reliability second subgroup comprises allocating the data fragments of the distributed chunk data structure to a second part of the medium reliability second subgroup and to the lower reliability third subgroup.
16. The non-transitory machine-readable medium of claim 15 , wherein the group of storage devices comprise solid state devices, and wherein the obtaining the reliability data comprises determining respective data writes to respective solid state devices.
17. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise erasure coding data of a chunk data structure to encoded data fragments and encoded coding fragments, writing the encoded data fragments to the data fragments allocated on the higher reliability first subgroup, and writing the encoded coding fragments to the coding fragments allocated on the medium reliability second subgroup.
18. The non-transitory machine-readable medium of claim 15 , wherein the determining the higher reliability first subgroup and the medium reliability second subgroup comprises sorting the group of storage devices based on the reliability data.
19. The non-transitory machine-readable medium of claim 15 , wherein the obtaining the reliability data comprises obtaining an age of a disk drive of the group of storage devices.
20. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise sorting the group of storage devices based on the reliability data to determine the higher reliability first subgroup and the medium reliability second subgroup.