Efficient erasure coding of large data objects
A system, computer program product, and computer-executable method for use with a distributed storage system comprising a plurality of storage nodes each having attached storage devices, the system, computer program product, and computer-executable method including receiving a request, at a first storage node of the plurality of storage nodes, to store a large portion of data, using at least one of a first type of data chunk and a plurality of a second type of data chunks to store the large portion of data, processing each of the plurality of the second type of data chunks, processing each of the at least one of the first type of data chunk, and returning an acknowledgement to the request.
1. A computer-executable method for use with a distributed storage system comprising a plurality of storage nodes each having attached storage devices, the computer-executable method comprising:
receiving a request, at a first storage node of the plurality of storage nodes, to store a large portion of data;
using at least one of a first type of data chunk and a plurality of a second type of data chunks to store the large portion of data
processing each of the plurality of the second type of data chunks by:
filling a first data chunk, of the plurality of the second type of data chunks, with a first portion of data from the large portion of data, wherein the first portion of data completely fills the first data chunk;
distributing the first data chunk to one of the plurality of storage nodes;
retaining, at the first storage node, the content of the first portion of data;
executing, on the first storage node, an erasure encoding task to generate coded fragments using the content of the first portion of data;
distributing the generated coded fragments to the plurality of storage nodes; and
deleting the content of the first portion of data from the first storage node;
processing each of the at least one of the first type of data chunk; and
returning an acknowledgement to the request.
2. The computer-executable method of claim 1 , wherein the returning an acknowledgment to the request is sent after each of the first type of data chunk and each of the second type of data chunks are protected.
3. The computer-executable method of claim 1 , wherein the large portion of data is represented by at least one of a first type of data chunk and a plurality of a second type of data chunk.
4. The computer-executable method of claim 1 , wherein the processing each of the at least one of the first type of data chunk comprises:
mirroring each of the at least one of the first type of data chunks.
5. The computer-executable method of claim 1 , wherein during the distribution of the second type of data chunk and associated generated coded fragments, each of the second type of data chunk and associated generated coded fragments are distributed to unique storage nodes of the plurality of storage nodes.
6. The computer-executable method of claim 1 , further comprising:
upon failure to protect any of the second type of data chunk created, returning a failure signal.
7. A system, comprising:
a distributed storage system including a plurality of storage nodes each having attached storage devices; and
computer-executable program logic encoded in memory of one or more computers enabled for use with the distributed storage system, wherein the computer-executable program logic is configured for the execution of:
receiving a request, at a first storage node of the plurality of storage nodes, to store a large portion of data;
using at least one of a first type of data chunk and a plurality of a second type of data chunks to store the large portion of data
processing each of the plurality of the second type of data chunks by:
filling a first data chunk, of the plurality of the second type of data chunks, with a first portion of data from the large portion of data, wherein the first portion of data completely fills the first data chunk;
distributing the first data chunk to one of the plurality of storage nodes;
retaining, at the first storage node, the content of the first portion of data;
executing, on the first storage node, an erasure encoding task to generate coded fragments using the content of the first portion of data;
distributing the generated coded fragments to the plurality of storage nodes; and
deleting the content of the first portion of data from the first storage node;
processing each of the at least one of the first type of data chunk; and
returning an acknowledgement to the request.
8. The system of claim 7 , wherein the returning an acknowledgment to the request is sent after each of the first type of data chunk and each of the second type of data chunks are protected.
9. The system of claim 7 , wherein the large portion of data is represented by at least one of a first type of data chunk and a plurality of a second type of data chunk.
10. The system of claim 7 , wherein the processing each of the at least one of the first type of data chunk comprises:
mirroring each of the at least one of the first type of data chunks.
11. The system of claim 7 , wherein during the distribution of the second type of data chunk and associated generated coded fragments, each of the second type of data chunk and associated generated coded fragments are distributed to unique storage nodes of the plurality of storage nodes.
12. The system of claim 7 , wherein the computer-executable program logic is further configured for the execution of:
upon failure to protect any of the second type of data chunk created, returning a failure signal.
13. A computer program product for use with a distributed storage system comprising a plurality of storage nodes each having attached storage devices, the computer program product comprising:
a non-transitory computer readable medium encoded with computer-executable code, the code configured to enable the execution of:
receiving a request, at a first storage node of the plurality of storage nodes, to store a large portion of data;
using at least one of a first type of data chunk and a plurality of a second type of data chunks to store the large portion of data
processing each of the plurality of the second type of data chunks by:
filling a first data chunk, of the plurality of the second type of data chunks, with a first portion of data from the large portion of data, wherein the first portion of data completely fills the first data chunk;
distributing the first data chunk to one of the plurality of storage nodes;
retaining, at the first storage node, the content of the first portion of data;
executing, on the first storage node, an erasure encoding task to generate coded fragments using the content of the first portion of data;
distributing the generated coded fragments to the plurality of storage nodes; and
deleting the content of the first portion of data from the first storage node;
processing each of the at least one of the first type of data chunk; and
returning an acknowledgement to the request.
14. The computer program product of claim 13 , wherein the returning an acknowledgment to the request is sent after each of the first type of data chunk and each of the second type of data chunks are protected.
15. The computer program product of claim 13 , wherein the large portion of data is represented by at least one of a first type of data chunk and a plurality of a second type of data chunk.
16. The computer program product of claim 13 , wherein the processing each of the at least one of the first type of data chunk comprises:
mirroring each of the at least one of the first type of data chunks.
17. The computer program product of claim 13 , wherein during the distribution of the second type of data chunk and associated generated coded fragments, each of the second type of data chunk and associated generated coded fragments are distributed to unique storage nodes of the plurality of storage nodes.