Systems and methods for variable-length chunking for deduplication
A computer-implemented method for variable-length chunking may include 1) identifying a first data stream subject to deduplication, 2) identifying a predetermined chunk of the first data stream that starts at a first location and ends at a second location within the first data stream, 3) identifying a second data stream with a matching chunk for the predetermined chunk that starts at a third location and ends at a fourth location within the second data stream, 4) identifying a subsequent chunk within the second data stream that starts at the fourth location and ends at a fifth location within the second data stream, 5) calculating a candidate boundary offset within the first data stream based on exceeding the second location by a difference between the fifth location and the fourth location, and 6) performing a boundary test at the candidate boundary offset. Various other methods, systems, and computer-readable media are disclosed.
1. A computer-implemented method for variable-length chunking for deduplication, at least a portion of the computer-implemented method being performed by a computing device comprising at least one processor, the computer-implemented method comprising:
identifying a first data stream subject to deduplication using variable-length chunking;
identifying a predetermined chunk of the first data stream, the predetermined chunk starting at a first location within the first data stream and ending at a second location within the first data stream;
identifying, based on the predetermined chunk, a second data stream that comprises a matching chunk that matches the predetermined chunk, the matching chunk starting at a third location within the second data stream and ending at a fourth location within the second data stream;
identifying a subsequent chunk within the second data stream, the subsequent chunk starting at the fourth location within the second data stream and ending at a fifth location within the second data stream;
calculating a candidate boundary offset within the first data stream based on exceeding the second location within the first data stream by a difference between the fifth location within the second data stream and the fourth location within the second data stream;
performing a boundary test at the candidate boundary offset to determine whether the candidate boundary offset is a location of a chunk boundary within the first data stream, wherein the boundary test indicates a likelihood that a candidate chunk beginning at the second location within the first data stream and ending at the candidate boundary offset matches the subsequent chunk; and
chunking the first data stream based on the boundary test by beginning a chunk at the second location and ending the chunk at the candidate boundary offset.
2. The computer-implemented method of claim 1 , wherein the fifth location was identified as a boundary of the subsequent chunk by performing the boundary test at the fifth location within the second data stream.
3. The computer-implemented method of claim 1 , wherein the identifying the second data stream comprises:
identifying a plurality of candidate data streams that comprise a plurality of matching chunks that match the predetermined chunk; and
selecting the second data stream from the plurality of candidate data streams based on an additional similarity between the first data stream and the second data stream.
4. The computer-implemented method of claim 3 , wherein selecting the second data stream based on the additional similarity comprises at least one of:
selecting the second data stream based on a similarity between a name of the first data stream and a name of the second data stream;
selecting the second data stream based on a similarity between a size of the first data stream and a size of the second data stream; and
selecting the second data stream based on a similarity between a relative location of the first location within the first data stream and a relative location of the third location within the second data stream.
5. The computer-implemented method of claim 1 ,
wherein performing the boundary test at the candidate boundary offset comprises determining that the candidate boundary offset does not comprise the chunk boundary within the first data stream; and
further comprising performing the boundary test at sequential offsets within the first data stream until the boundary test identifies the chunk boundary within the first data stream.
6. The computer-implemented method of claim 1 , wherein the identifying the predetermined chunk comprises the performing the boundary test at sequential offsets within the first data stream until the boundary test identifies the second location within the first data stream as a boundary of the predetermined chunk.
7. The computer-implemented method of claim 1 , wherein the identifying the predetermined chunk comprises:
identifying a candidate data stream with metadata that matches the first data stream;
identifying a first chunk end boundary within the candidate data stream; and
performing the boundary test at an offset within the first data stream equal to an offset of the first chunk end boundary within the candidate data stream to determine whether the offset within the first data stream is a location of a boundary of the predetermined chunk.
8. The computer-implemented method of claim 1 , wherein the performing the boundary test comprises calculating a checksum at the candidate boundary offset.
9. The computer-implemented method of claim 1 ,
wherein the performing the boundary test at the candidate boundary offset comprises determining that the candidate boundary offset comprises the chunk boundary within the first data stream; and
further comprising identifying a new chunk beginning at the second location within the first data stream and ending at the candidate boundary offset within the first data stream.
10. The computer-implemented method of claim 9 , further comprising deduplicating the first data stream by deduplicating the new chunk with at least one identical chunk.
11. A system for variable-length chunking for deduplication,
the system comprising:
an identification module programmed to:
identify a first data stream subject to deduplication using variable-length chunking;
identify a predetermined chunk of the first data stream, the predetermined chunk starting at a first location within the first data stream and ending at a second location within the first data stream;
a matching module programmed to identify, based on the predetermined chunk, a second data stream that comprises a matching chunk that matches the predetermined chunk, the matching chunk starting at a third location within the second data stream and ending at a fourth location within the second data stream;
a location module programmed to identify a subsequent chunk within the second data stream, the subsequent chunk starting at the fourth location within the second data stream and ending at a fifth location within the second data stream;
a mapping module programmed to calculate a candidate boundary offset within the first data stream based on exceeding the second location within the first data stream by a difference between the fifth location within the second data stream and the fourth location within the second data stream;
a testing module programmed to:
perform a boundary test at the candidate boundary offset to determine whether the candidate boundary offset is a location of a chunk boundary within the first data stream, wherein the boundary test indicates a likelihood that a candidate chunk beginning at the second location within the first data stream and ending at the candidate boundary offset matches the subsequent chunk;
chunk the first data stream based on the boundary test by beginning a chunk at the second location and ending the chunk at the candidate boundary offset; and
at least one processor configured to execute the identification module, the matching module, the location module, the mapping module, and the testing module.
12. The system of claim 11 , wherein the fifth location was identified as a boundary of the subsequent chunk by performing the boundary test at the fifth location within the second data stream.
13. The system of claim 11 , wherein the matching module is programmed to identify the second data stream by:
identifying a plurality of candidate data streams that comprise a plurality of matching chunks that match the predetermined chunk; and
selecting the second data stream from the plurality of candidate data streams based on an additional similarity between the first data stream and the second data stream.
14. The system of claim 13 , wherein the matching module is programmed to select the second data stream based on the additional similarity by at least one of:
selecting the second data stream based on a similarity between a name of the first data stream and a name of the second data stream;
selecting the second data stream based on a similarity between a size of the first data stream and a size of the second data stream; and
selecting the second data stream based on a similarity between a relative location of the first location within the first data stream and a relative location of the third location within the second data stream.
15. The system of claim 11 ,
wherein the testing module is programmed to perform the boundary test at the candidate boundary offset by determining that the candidate boundary offset does not comprise the chunk boundary within the first data stream; and
wherein the testing module is further programmed to perform the boundary test at sequential offsets within the first data stream until the boundary test identifies the chunk boundary within the first data stream.
16. The system of claim 11 , wherein the identification module is programmed to identify the predetermined chunk by performing the boundary test at sequential offsets within the first data stream until the boundary test identifies the second location within the first data stream as a boundary of the predetermined chunk.
17. The system of claim 11 , wherein the identification module is programmed to identify the predetermined chunk by:
identifying a candidate data stream with metadata that matches the first data stream;
identifying a first chunk end boundary within the candidate data stream; and
performing the boundary test at an offset within the first data stream equal to an offset of the first chunk end boundary within the candidate data stream to determine whether the offset within the first data stream is a location of a boundary of the predetermined chunk.
18. The system of claim 11 , wherein the testing module is programmed to perform the boundary test by calculating a checksum at the candidate boundary offset.
19. The system of claim 11 ,
wherein the testing module is programmed to perform the boundary test at the candidate boundary offset by determining that the candidate boundary offset comprises the chunk boundary within the first data stream;
further comprising a deduplication module programmed to:
identify a new chunk beginning at the second location within the first data stream and ending at the candidate boundary offset within the first data stream; and
deduplicate the first data stream by deduplicating the new chunk with at least one identical chunk.
20. A non-transitory computer-readable-storage medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
identify a first data stream subject to deduplication using variable-length chunking;
identify a predetermined chunk of the first data stream, the predetermined chunk starting at a first location within the first data stream and ending at a second location within the first data stream;
identify, based on the predetermined chunk, a second data stream that comprises a matching chunk that matches the predetermined chunk, the matching chunk starting at a third location within the second data stream and ending at a fourth location within the second data stream;
identify a subsequent chunk within the second data stream, the subsequent chunk starting at the fourth location within the second data stream and ending at a fifth location within the second data stream;
calculate a candidate boundary offset within the first data stream based on exceeding the second location within the first data stream by a difference between the fifth location within the second data stream and the fourth location within the second data stream;
perform a boundary test at the candidate boundary offset to determine whether the candidate boundary offset is a location of a chunk boundary within the first data stream, wherein the boundary test indicates a likelihood that a candidate chunk beginning at the second location within the first data stream and ending at the candidate boundary offset matches the subsequent chunk; and
chunk the first data stream based on the boundary test by beginning a chunk at the second location and ending the chunk at the candidate boundary offset.