Sub-block deduplication using sector hashing

1. A method of performing deduplication, the method comprising:

selecting, by applying a deterministic selection criterion, a sub-block of a block of data that contains multiple sub-blocks;

performing a lookup, into a deduplication table, of a digest generated by hashing the selected sub-block, the lookup matching an entry indexed by the digest in the deduplication table, the entry identifying a previously processed block;

effecting storage of the block, including pointing to the previously processed block; and

for another block of data that also contains multiple sub-blocks:

selecting, by applying the deterministic selection criterion, another sub-block of the other block of data;

performing another lookup, into the deduplication table, of another digest generated by hashing the other selected sub-block;

failing to find any entry indexed by the other digest in the deduplication table, and, in response, looking up, in the deduplication table, a block digest generated by hashing the other block in its entirety; and

failing to find any entry indexed by the block digest in the deduplication table, and, in response:

writing contents of the other block to persistent storage; and

inserting another entry, indexed by the other digest, into the deduplication table, the other entry identifying the other block as having been processed.

2. The method of claim 1 ,

wherein the method further comprises identifying a target range of the previously processed block whose contents match a corresponding range of the block of data; and

wherein pointing to the previously processed block includes storing metadata in connection with the block that associates the identified target range as being part of the block of data.

3. The method of claim 2 wherein the previously processed block also includes a unique range that does not match contents of the block of data.

4. The method of claim 3 ,

wherein the entry further identifies a first offset within the previously processed block, the first offset providing a location of contents within the previously processed block that are shared with the selected sub-block;

wherein the selected sub-block is located at a second offset within the block of data; and

wherein identifying the target range includes comparing the second offset with the first offset, and determining that the second offset differs from the first offset.

5. The method of claim 3 , wherein identifying the target range includes:

generating a first block digest by hashing the block in its entirety;

comparing the first block digest with a previously-stored second block digest that was generated by hashing the previously processed block in its entirety, and determining that the first block digest differs from the second block digest.

6. The method of claim 3 ,

wherein the entry further identifies a first offset within the previously processed block, the first offset providing a location of a prime sub-block within the previously processed block whose contents are shared with the selected sub-block;

wherein the selected sub-block is located at a second offset within the block of data; and

wherein identifying the target range includes:

identifying the prime sub-block as part of the target range; and

comparing a first adjacent sub-block of the previously processed block having an offset adjacent to the first offset with a second adjacent sub-block of the block of data having an offset adjacent to the second offset to identify whether the first adjacent sub-block is also part of the target range.

7. The method of claim 6 ,

wherein identifying the target range further includes:

working outwards from the prime sub-block, comparing additional sub-blocks of the previously processed block to corresponding sub-blocks of the block of data until comparison fails; and

identifying each additional sub-block of the previously processed block for which comparison to a corresponding sub-block of the block of data succeeded as part of the target range; and

wherein effecting storage of the block further includes writing contents of the block of data that are not also included within the identified target range to persistent storage.

8. The method of claim 6 ,

wherein identifying the target range further includes:

successfully comparing additional sub-blocks of the previously processed block up to an end of the previously processed block with corresponding sub-blocks of the block of data; and

identifying each additional sub-block of the previously processed block for which comparison to a corresponding sub-block of the block of data succeeded as part of the target range;

wherein the method further includes:

identifying an adjacent previously processed block located adjacent to the previously processed block within a logical address space;

comparing additional sub-blocks of the adjacent previously processed block with corresponding sub-blocks of the block of data; and

identifying each additional sub-block of the adjacent previously processed block for which comparison to a corresponding sub-block of the block of data succeeded as part of another target range whose contents are included within the block of data; and

wherein pointing to the previously processed block further includes storing metadata in connection with the block that associates the identified other target range as being part of the block of data.

9. The method of claim 8 ,

wherein all sub-blocks of the block of data compare successfully to corresponding sub-blocks of the previously processed block and the adjacent previously processed block; and

wherein effecting storage of the block further includes not writing any portion of the block to persistent storage.

10. The method of claim 3 wherein the method further comprises:

generating another block digest by hashing the block in its entirety; and

inserting another entry, indexed by the other block digest, into the deduplication table, the other entry identifying the block as having been processed.

11. The method of claim 2 wherein contents of the block of data in its entirety are identical to the contents of the identified target range.

12. The method of claim 11 , wherein identifying the target range includes:

generating a first block digest by hashing the block in its entirety; and

comparing the first block digest to a previously-stored second block digest that was generated by hashing the previously processed block in its entirety, and in response, determining that the first block digest equals the second block digest.

13. The method of claim 1 , wherein selecting includes:

calculating an entropy of each of the multiple sub-blocks of the block; and

selecting a sub-block from the multiple sub-blocks having a highest calculated entropy as the selected sub-block.

14. The method of claim 13 wherein the method further comprises, for yet another block of data that also contains multiple sub-blocks:

calculating an entropy of each of the multiple sub-blocks of the yet other block;

selecting another sub-block from the multiple sub-blocks of the yet other block having a highest calculated entropy as a candidate sub-block;

comparing the highest calculated entropy of the candidate sub-block to a predetermined threshold value, yielding a threshold result; and

in response to the comparison yielding, as its threshold result, a determination that the highest calculated entropy of the candidate sub-block is less than the predetermined threshold value, looking up, in the deduplication table, yet another digest generated by hashing the yet other block in its entirety.

15. The method of claim 14 wherein the method further comprises, in response to looking up, in the deduplication table, the yet other digest generated by hashing the yet other block in its entirety:

failing to find any entry indexed by the yet other digest in the deduplication table;

writing contents of the yet other block to persistent storage; and

inserting yet another entry indexed by the yet other digest into the deduplication table, the yet other entry identifying the yet other block as having been processed.

16. The method of claim 14 wherein the method further comprises, in response to looking up, in the deduplication table, the yet other digest generated by hashing the yet other block in its entirety:

finding yet another entry indexed by the yet other digest in the deduplication table, the yet other entry identifying another previously processed block; and

effecting storage of the yet other block by pointing to the previously processed block and not writing any portion of the yet other block to persistent storage.

17. An apparatus comprising processing circuitry coupled to memory configured to:

select, by applying a deterministic selection criterion, a sub-block of a block of data that contains multiple sub-blocks, wherein selecting includes:

calculating an entropy of each of the multiple sub-blocks of the block; and

selecting a sub-block from the multiple sub-blocks having a highest calculated entropy as the selected sub-block;

perform a lookup, into a deduplication table, of a digest generated by hashing the selected sub-block, the lookup matching an entry indexed by the digest in the deduplication table, the entry identifying a previously processed block; and

effect storage of the block, including pointing to the previously processed block; and

for another block of data that also contains multiple sub-blocks:

selecting, by applying the deterministic selection criterion, another sub-block of the other block of data;

performing another lookup, into the deduplication table, of another digest generated by hashing the other selected sub-block;

failing to find any entry indexed by the other digest in the deduplication table, and, in response, looking up, in the deduplication table, a block digest generated by hashing the other block in its entirety; and

failing to find any entry indexed by the block digest in the deduplication table, and, in response:

writing contents of the other block to persistent storage; and

inserting another entry, indexed by the other digest, into the deduplication table, the other entry identifying the other block as having been processed.

18. A computer program product comprising a non-transitory computer-readable storage medium storing a set of instructions, which, when executed by a computing device cause the computing device to perform deduplication by:

selecting, by applying a deterministic selection criterion, a sub-block of a block of data that contains multiple sub-blocks;

performing a lookup, into a deduplication table, of a digest generated by hashing the selected sub-block, the lookup matching an entry indexed by the digest in the deduplication table, the entry identifying a previously processed block; and

effecting storage of the block, including pointing to the previously processed block; and

for another block of data that also contains multiple sub-blocks:

selecting, by applying the deterministic selection criterion, another sub-block of the other block of data;

performing another lookup, into the deduplication table, of another digest generated by hashing the other selected sub-block;

failing to find any entry indexed by the other digest in the deduplication table, and, in response, looking up, in the deduplication table, a block digest generated by hashing the other block in its entirety; and

failing to find any entry indexed by the block digest in the deduplication table, and, in response:

writing contents of the other block to persistent storage; and

inserting another entry, indexed by the other digest, into the deduplication table, the other entry identifying the other block as having been processed.