Opportunistic partial deduplication
A technique for managing deduplication performs partial-block matching opportunistically by leveraging information acquired during times when a storage system has available resources. The information identifies anchor blocks that are likely targets for partial-block matches, based on discovering that the anchor blocks belong to populations of blocks that have high similarity. When processing write requests, inline activities access anchor blocks that closely match newly arriving candidate blocks and perform partial-block deduplication against those anchor blocks.
1. A method of performing data deduplication, the method comprising:
forming populations of data blocks by (i) performing clustering of data blocks based on similarity among data blocks, the clustering producing a plurality of clusters, and (ii) selecting, based on cluster size, the populations of data blocks as a subset of the plurality of clusters;
identifying multiple target blocks that belong to respective populations of data blocks, each population including data blocks that partially match one another;
upon obtaining a candidate block, selecting one of the target blocks based on that target block partially matching the candidate block; and
effectuating storage of the candidate block at least in part by providing a reference to the selected target block.
2. The method of claim 1 , further comprising:
storing digests of the target blocks in memory; and
performing inline deduplication based on creating a digest of a candidate block and finding in memory a digest of a target block that partially matches the digest of the candidate block.
3. The method of claim 2 , wherein storing the digests of the target blocks in memory includes providing the digests in an anchor table in memory, the anchor table associating the digests of target blocks with respective storage locations of the target blocks.
4. The method of claim 2 , wherein the digests of the target blocks and the digest of the candidate block include similarity hashes (sim hashes), the sim hashes of different blocks differing from one another in relation to differences among the different blocks.
5. The method of claim 4 , wherein performing the inline deduplication includes determining that the digest of the candidate block and a digest of a target block differ by no more than a specified Hamming distance.
6. The method of claim 1 , wherein performing the clustering of data blocks based on similarity includes:
generating similarity hashes (sim hashes) of the data blocks; and
comparing sim hashes generated from the data blocks, the sim hashes of the data blocks differing from one another relation to differences among the data blocks from which the sim hashes are generated.
7. The method of claim 6 , wherein performing the clustering of data blocks based on similarity further includes grouping together data blocks whose sim hashes differ from one another by less than a specified Hamming distance.
8. The method of claim 6 , wherein identifying the target blocks that belong to the respective populations includes selecting a representative block from each of the subset of clusters as the respective target block.
9. The method of claim 8 , wherein selecting each representative block includes choosing a block from each of the subset of clusters whose sim hash minimizes Hamming distances to sim hashes of other blocks in the respective subset of clusters.
10. The method of claim 1 , wherein effectuating storage of the candidate block further includes storing difference data between the candidate block and the selected target block.
11. A computerized apparatus, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
form populations of data blocks by (i) performing clustering of data blocks based on similarity among data blocks, the clustering producing a plurality of clusters, and (ii) selecting, based on duster size, the populations of data blocks as a subset of the plurality of clusters;
identify multiple target blocks that belong to respective populations of data blocks, each population including data blocks that partially match one another;
upon obtaining a candidate block, select one of the target blocks based on that target block partially matching the candidate block; and
effectuate storage of the candidate block at least in part by providing a reference to the selected target block.
12. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of performing data deduplication, the method comprising:
forming populations of data blocks by (i) performing clustering of data blocks based on similarity among data blocks, the clustering producing a plurality of clusters, and (ii) selecting, based on cluster size, the populations of data blocks as a subset of the plurality of clusters;
identifying multiple target blocks that belong to respective populations of data blocks, each population including data blocks that partially match one another;
upon obtaining a candidate block, selecting one of the target blocks based on that target block partially matching the candidate block; and
effectuating storage of the candidate block at least in part by providing a reference to the selected target block.
13. The computer program product of claim 12 , wherein the method further comprises:
storing digests of the target blocks in memory; and
performing inline deduplication based on creating a digest of a candidate block and finding in memory a digest of a target block that partially matches the digest of the candidate block.
14. The computer program product of claim 13 , wherein storing the digests of the target blocks in memory includes providing the digests in an anchor table in memory, the anchor table associating the digests of target blocks with respective storage locations of the target blocks.
15. The computer program product of claim 13 , wherein the digests of the target blocks and the digest of the candidate block include similarity hashes (sim hashes), the sim hashes of different blocks differing from one another in relation to differences among the different blocks.
16. The computer program product of claim 15 , wherein performing the inline deduplication includes determining that the digest of the candidate block and a digest of a target block differ by no more than a specified Hamming distance.
17. The computer program product of claim 12 , wherein performing the clustering of data blocks based on similarity includes:
generating similarity hashes (sim hashes) of the data blocks; and
comparing sim hashes generated from the data blocks, the sim hashes of the data blocks differing from one another relation to differences among the data blocks from which the sim hashes are generated.