Functional caching in erasure coded storage
Encoding a file into a plurality of chunks, wherein a subset of the plurality of chunks may be used to create a functional equivalent of the file. At least one additional chunk is created from the plurality of chunks. The at least one additional chunk is directed to be stored in a cache memory and the plurality of chunks are directed to be stored on at least one storage node. Upon demand for the file, at least one additional chunk is cased to be retrieved from the cache and at least a portion of the plurality of chunks are caused to be retrieved from the at least one storage node and the functional equivalent of the file is constructed through utilization of the at least one additional chunk and the portion of the plurality of chunks.
1. An apparatus comprising:
a processor; and
memory coupled to the processor, the memory comprising executable instructions that cause the processor to effectuate operations comprising:
encoding a file into a plurality n of chunks, wherein a subset k of the n chunks are used to create a functional equivalent of the file and k<n;
creating at least one additional chunk d from the plurality of n chunks, wherein k of the n chunks and the at least one additional chunk d is used to create the functional equivalent of the file, wherein d is greater than or equal to one;
directing the at least one additional chunk d to be stored only in a cache memory and the n chunks to be stored only in non-cache memory on at least one storage node;
upon demand for the file, causing the at least one additional chunk d to be retrieved from the cache memory and at least one of the n chunks from the non-cache memory; and
constructing the functional equivalent of the file through utilization of the at least one additional chunk d and the at least one of the n chunks.
2. The apparatus of claim 1 , wherein the non-cache memory comprises
at least one storage node.
3. The apparatus of claim 2 , wherein the at least one storage node comprises a plurality of storage nodes and the operation of directing comprises:
utilizing the n chunks to create the d chunks, wherein d>1;
directing each of the d chunks to be stored in the cache memory; and
directing each of the n chunks to be stored on one of the storage nodes.
4. The apparatus of claim 3 , wherein the operation of causing comprises requesting each one of the storage nodes to transmit a respective one of the n chunks.
5. The apparatus of claim 4 , wherein the operation of constructing comprises constructing the functional equivalent upon receipt of a sufficient number of the n chunks and d chunks to create the functional equivalent of the file.
6. The apparatus of claim 1 , wherein the operation of encoding comprises encoding the file through employment of a maximum distance separable (MDS) erasure code to create a (n, k) erasure code.
7. The apparatus of 6 , wherein the operations further comprise:
using the d chunk(s) to create a (n+d, k) erasure code.
8. The apparatus of claim 7 , wherein the operations further comprise:
receiving d chunks from the cache; and
receiving k-d chunks from the at least one storage node.
9. The apparatus of claim 1 , wherein the operations further comprise:
receiving a plurality of requests for the file;
calculating a rate at which the plurality of requests have been received;
using the rate in a determination to run a cache optimization routine.
10. The apparatus of claim 1 , wherein creating is based on an output of the cache optimization routine.
11. A method operating on a server device comprising:
encoding a file into a plurality n of chunks, wherein a subset k of the n chunks are used to create a functional equivalent of the file and k<n;
creating at least one additional chunk d from the plurality of n chunks, wherein k of the n chunks and the at least one additional chunk d is used to create the functional equivalent of the file, wherein d is greater than or equal to one;
directing the at least one additional chunk d to be stored only in a cache memory and the n chunks to be stored only in non-cache memory on at least one storage node;
upon demand for the file, causing the at least one additional chunk d to be retrieved from the cache memory and at least one of the n chunks from the non-cache memory; and
constructing the functional equivalent of the file through utilization of the at least one additional chunk d and the at least one of the n chunks.
12. The method of claim 11 , further comprising:
upon demand for the file, causing the at least one of the n chunks to be retrieved from the at least one storage node.
13. The method of claim 12 , wherein the at least one storage node comprises a plurality of storage nodes and the step of directing comprises:
utilizing the n chunks to create the d chunks, wherein d>1;
directing each of the d chunks to be stored in the cache memory; and
directing each of the n chunks to be stored on one of the storage nodes.
14. The method of claim 13 , wherein the step of causing comprises requesting each one of the storage nodes to transmit a respective one of the n chunks.
15. The method of claim 14 , wherein the step of constructing comprises constructing the functional equivalent upon receipt of a sufficient number of the n chunks and d chunks to create the functional equivalent of the file.
16. The method of claim 11 , wherein the step of encoding comprises encoding the file through employment of a maximum distance separable (MDS) erasure code to create a (n, k) erasure code.
17. The method of claim 16 , further comprising:
using the the d chunk(s) to create a (n+d, k) erasure code.
18. The method of claim 17 , further comprising:
receiving d chunks from the cache; and
receiving k-d chunks from the at least one storage node.
19. The method of claim 11 , further comprising:
receiving a plurality of requests for the file;
calculating a rate at which the plurality of requests have been received;
using the rate in a determination to run a cache optimization routine.
20. The method of claim 11 , wherein the step of creating is based on an output of the cache optimization routine.