Systems and methods for parallel content-defined data chunking
A computer-implemented method for parallel content-defined data chunking may include (1) identifying a data stream to be chunked, (2) splitting the data stream into a plurality of data sub-streams by alternatingly dividing consecutive bytes of the data stream among the plurality of data sub-streams, and (3) chunking, in parallel, each data sub-stream within the plurality of data sub-streams into a plurality of data segments using a content-defined chunking algorithm. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for parallel content-defined data chunking, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
identifying a data stream to be chunked;
creating, from the data stream, a plurality of data sub-streams by alternatingly dividing bytes of the data stream among the plurality of data sub-streams such that:
each of the plurality of data sub-streams comprises two or more of the bytes of the data stream;
consecutive bytes within the data stream are contained in different data sub-streams within the plurality of data sub-streams;
chunking, in parallel, each data sub-stream within the plurality of data sub-streams into a plurality of data segments using a content-defined chunking algorithm.
2. The computer-implemented method of claim 1 , further comprising storing the data stream by individually storing each data segment within the plurality of data segments of each data sub-stream within the plurality of data sub-streams.
3. The computer-implemented method of claim 2 , further comprising:
receiving, after storing the data stream, a request for the data stream;
identifying, in response to the request, the plurality of data segments of each data sub-stream within the plurality of data sub-streams;
reconstructing each data sub-stream within the plurality of data sub-streams from the plurality of data segments of each data sub-stream within the plurality of data sub-streams;
reconstructing the data stream from the plurality of reconstructed data sub-streams;
responding to the request for the data stream with the reconstructed data stream.
4. The computer-implemented method of claim 2 , wherein storing the data stream by individually storing each data segment within the plurality of data segments of each data sub-stream within the plurality of data sub-streams comprises storing only unique data segments within the plurality of data segments of each data sub-stream within the plurality of data sub-streams.
5. The computer-implemented method of claim 1 , wherein creating the plurality of data sub-streams comprises alternatingly dividing the bytes of the data stream among a plurality of data buffers, wherein each data buffer within the plurality of data buffers has a size equal to the maximum size of data segments identified by the content-defined chunking algorithm.
6. The computer-implemented method of claim 1 , wherein chunking each data sub-stream within the plurality of data sub-streams comprises beginning to chunk at least one data sub-stream within the plurality of data sub-streams when one segment marker could be identified within the data sub-stream.
7. The computer-implemented method of claim 1 , wherein each data sub-stream within the plurality of data sub-streams is chunked by a separate thread of execution.
8. The computer-implemented method of claim 1 , wherein:
the data stream comprises at least one file that is to be deduplicated;
the file comprises the consecutive bytes.
9. A system for parallel content-defined data chunking, the system comprising:
an identifying module that identifies a data stream to be chunked;
a splitting module that creates, from the data stream, a plurality of data sub-streams by alternatingly dividing bytes of the data stream among the plurality of data sub-streams such that:
each of the plurality of data sub-streams comprises two or more of the bytes of the data stream;
consecutive bytes within the data stream are contained in different data sub-streams within the plurality of data sub-streams;
a chunking module that chunks, in parallel, each data sub-stream within the plurality of data sub-streams into a plurality of data segments using a content-defined chunking algorithm;
at least one processor configured to execute the identifying module, the splitting module, and the chunking module.
10. The system of claim 9 , further comprising a storing module that stores the data stream by individually storing each data segment within the plurality of data segments of each data sub-stream within the plurality of data sub-streams.
11. The system claim 10 , wherein the storing module further:
receives, after storing the data stream, a request for the data stream;
identifies, in response to the request, the plurality of data segments of each data sub-stream within the plurality of data sub-streams;
reconstructs each data sub-stream within the plurality of data sub-streams from the plurality of data segments of each data sub-stream within the plurality of data sub-streams;
reconstructs the data stream from the plurality of reconstructed data sub-streams;
responds to the request for the data stream with the reconstructed data stream.
12. The system of claim 10 , wherein the storing module stores the data stream by individually storing each data segment within the plurality of data segments of each data sub-stream within the plurality of data sub-streams by storing only unique data segments within the plurality of data segments of each data sub-stream within the plurality of data sub-streams.
13. The system of claim 9 , wherein the splitting module creates the plurality of data sub-streams by alternatingly dividing the bytes of the data stream among a plurality of data buffers, wherein each data buffer within the plurality of data buffers has a size equal to the maximum size of data segments identified by the content-defined chunking algorithm.
14. The system of claim 9 , wherein the chunking module chunks each data sub-stream within the plurality of data sub-streams by beginning to chunk at least one data sub-stream within the plurality of data sub-streams when one segment marker could be identified within the data sub-stream.
15. The system of claim 9 , wherein each data sub-stream within the plurality of data sub-streams is chunked by a separate thread of execution.
16. The system of claim 9 , wherein:
the data stream comprises at least one file that is to be deduplicated;
the file comprises the consecutive bytes.
17. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
identify a data stream to be chunked;
create, from the data stream, a plurality of data sub-streams by alternatingly dividing consecutive bytes of the data stream among the plurality of data sub-streams such that:
each of the plurality of data sub-streams comprises two or more of the bytes of the data stream;
consecutive bytes within the data stream are contained in different data sub-streams within the plurality of data sub-streams;
chunk, in parallel, each data sub-stream within the plurality of data sub-streams into a plurality of data segments using a content-defined chunking algorithm.
18. The non-transitory computer-readable medium of claim 17 , wherein the one or more computer-executable instructions, when executed by the processor of the computing device, further cause the computing device to store the data stream by individually storing each data segment within the plurality of data segments of each data sub-stream within the plurality of data sub-streams.
19. The non-transitory computer-readable medium of claim 18 , wherein
the one or more computer-executable instructions, when executed by the processor of the computing device, further cause the computing device to:
receive, after storing the data stream, a request for the data stream;
identify, in response to the request, the plurality of data segments of each data sub-stream within the plurality of data sub-streams;
reconstruct each data sub-stream within the plurality of data sub-streams from the plurality of data segments of each data sub-stream within the plurality of data sub-streams;
reconstruct the data stream from the plurality of reconstructed data sub-streams;
respond to the request for the data stream with the reconstructed data stream.
20. The non-transitory computer-readable medium of claim 18 , wherein the one or more computer-executable instructions, when executed by the processor of the computing device, cause the computing device to store the data stream by individually storing each data segment within the plurality of data segments of each data sub-stream within the plurality of data sub-streams by causing the computing device to store only unique data segments within the plurality of data segments of each data sub-stream within the plurality of data sub-streams.