Systems and methods for caching in an erasure-coded system
The disclosed computer-implemented method for caching in an erasure-coded system may include partitioning data into data chunks and storing the data chunks across at least two data storage devices. The method may also include encoding the data chunks into at least one parity chunk and storing the at least one parity chunk on at least one parity storage device. The method may further include caching (parity and/or data functions) to at least one storage device, when a threshold value is reached. The method may include detecting a storage failure on one of the data storage devices and subsequent thereto, prioritizing the caching of certain functions over those of others. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for caching in an erasure-coded system, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
partitioning data into a plurality of data chunks and storing the plurality of partitioned data chunks across at least two data storage devices;
encoding the plurality of partitioned data chunks into at least one parity chunk and storing the at least one parity chunk on at least one parity storage device;
in a normal mode:
for each of the at least two data storage devices, caching data reads to the data storage device when a number of reads to the data storage device reaches a data read threshold; and
for the parity storage device, caching local parity writes to the parity storage device when a number of local parity writes reaches a parity threshold value and prioritizing caching parity writes that include reads and writes over parity writes that do not include reads; and
in a degraded mode, initiated in response to detecting a storage failure on one of the at least two data storage devices:
for the at least two data storage devices, prioritizing the caching of the data reads to the data storage device that failed over data storage devices that did not fail; and
for the parity storage device, continuing to cache local parity writes to the parity storage device when a number of local parity writes reaches a parity threshold value and prioritize caching parity writes that include reads and writes over parity writes that do not include reads.
2. The computer-implemented method according to claim 1 , further comprising:
identifying a plurality of local parity writes to the parity storage device;
determining that a number of the local parity writes reaches the parity threshold value; and
in response to determining that the number of the local parity writes reaches the parity threshold value, caching the local parity writes to the parity storage device.
3. The computer-implemented method according to claim 2 , further comprising:
prioritizing the caching of a parity write corresponding to a partial stripe write to the parity storage device over the caching of a parity write that does not involve a parity read.
4. The computer-implemented method according to claim 1 , wherein the at least two data storage devices comprise more than two data storage devices.
5. The computer-implemented method according to claim 4 , further comprising caching the data reads to two of the at least two data storage devices.
6. The computer-implemented method according to claim 1 , further comprising encoding the plurality of partitioned data chunks into a plurality of parity chunks and storing the plurality of parity chunks on a plurality of parity storage devices.
7. The computer-implemented method according to claim 1 , wherein at least one of the at least two data storage devices and the at least one parity storage device comprise a same storage device.
8. A system for caching in an erasure-coded system, which includes at least a cache and a plurality of storage devices, the system comprising:
a partitioning module, stored in memory, that partitions data into a plurality of data chunks and stores the plurality of partitioned data chunks across at least two data storage devices;
an encoding module, stored in memory, that encodes the plurality of partitioned data chunks into at least one parity chunk and stores the at least one parity chunk on at least one parity storage device;
a caching module, stored in memory, that:
in a normal mode:
for each of the at least two data storage devices, caches data reads to the data storage device when a number of reads to the data storage device reaches a data read threshold; and
for the parity storage device, caches local parity writes to the parity storage device when a number of local parity writes reaches a parity threshold value and prioritizes caching parity writes that include reads and writes over parity writes that do not include reads; and
in a degraded mode, initiated in response to detecting a storage failure on one of the at least two data storage devices:
for the at least two data storage devices, prioritizes the caching of the data reads to the data storage device that failed over data storage devices that did not fail; and
for the parity storage device, continues to cache local parity writes to the parity storage device when a number of local parity writes reaches a parity threshold value and prioritize caching parity writes that include reads and writes over parity writes that do not include reads.
9. The system according to claim 8 , wherein the caching module is further configured to:
identify a plurality of local parity writes to the parity storage device;
determine that a number of the local parity writes reaches the parity threshold value; and
in response to determining that the number of the local parity writes reaches the parity threshold value, cache the local parity writes to the parity storage device.
10. The system according to claim 9 , wherein the caching module prioritizes caching of a parity write corresponding to a partial stripe write to the at least one parity storage device over a parity write which does not involve a parity read.
11. The system according to claim 10 , wherein the data caching module caches a plurality of data reads to two of the at least two data storage devices.
12. The system according to claim 8 , wherein the partitioning module stores the plurality of partitioned data chunks on more than two data storage devices.
13. The system according to claim 8 , wherein the encoding module encodes the plurality of partitioned data chunks into a plurality of parity chunks and stores the plurality of parity chunks on a plurality of parity storage devices.
14. The system according to claim 8 , wherein the partitioning module and the encoding module respectively store the plurality of partitioned data chunks and the at least one parity chunk on a common storage device.
15. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
partition data into a plurality of data chunks and store the plurality of partitioned data chunks across at least two data storage devices;
encode the plurality of partitioned data chunks into at least one parity chunk and store the at least one parity chunk on at least one parity storage device;
in a normal mode:
for each of the at least two data storage devices, cache data reads to the data storage device when a number of reads to the data storage device reaches a data read threshold; and
for the at least one parity storage device, cache local parity writes to the parity storage device when a number of local parity writes reaches a parity threshold value and prioritize caching parity writes that include reads and writes over parity writes that do not include reads; and
in a degraded mode, initiated in response to detecting a storage failure on one of the at least two data storage devices:
for the at least two data storage devices, prioritize the caching of the data reads to the data storage device that failed over data storage devices that did not fail; and
for the at least one parity storage device, continue to cache local parity writes to the parity storage device when a number of local parity writes reaches a parity threshold value and prioritize caching parity writes that include reads and writes over parity writes that do not include reads.
16. The non-transitory computer-readable medium according to claim 15 , wherein the instructions further cause the computing device to:
identify a plurality of local parity writes to the parity storage device;
determine that a number of the local parity writes reaches the parity threshold value; and
in response to determining that the number of the local parity writes reaches the parity threshold value, cache the local parity writes to the parity storage device.
17. The non-transitory computer-readable medium according to claim 16 , wherein the one or more computer-executable instructions cause the computing device to cache a plurality of data reads on two of the at least two data storage devices.
18. The non-transitory computer-readable medium according to claim 15 , wherein the instructions further cause the computing device to prioritize caching of a parity write corresponding to a partial stripe write to the at least one parity storage device over a parity write which does not involve a parity read.
19. The non-transitory computer-readable medium according to claim 15 , wherein the one or more computer-executable instructions cause the computing device to encode the plurality of partitioned data chunks into a plurality of parity chunks and store the plurality of parity chunks on a plurality of parity storage devices.
20. The non-transitory computer-readable medium according to claim 15 , wherein the one or more computer-executable instructions cause the computing device to store the plurality of partitioned data chunks and the at least one parity chunk on a common storage device.