Extending deduplication matches using data comparison
A technique for performing deduplication extends a match found in a digest database by looking forward and/or backward to adjacent data elements. The technique performs data comparisons between data blocks adjacent to a candidate block and corresponding data blocks adjacent to an identified target block. If the data comparisons indicate that an adjacent candidate block matches an adjacent target block, then the adjacent candidate block may be deduplicated to the adjacent target block without having to compute a hash value of the adjacent candidate block or to perform a separate lookup into the digest database.
1. A method of performing data deduplication, the method comprising:
accessing a candidate block and a set of adjacent candidate blocks adjacent to the candidate block;
in response to finding a matching entry in a database to the candidate block, accessing a target block referenced by the matching entry in the database and accessing a set of adjacent target blocks adjacent to the target block;
performing a data comparison between at least one of the set of adjacent candidate blocks and at least one of the set of adjacent target blocks; and
in response to the data comparison indicating one or more matches, effectuating storage of each matched adjacent candidate block at least in part by providing a reference to the respective matching adjacent target block,
wherein performing the data comparison includes determining whether a first adjacent candidate block immediately adjacent to the candidate block matches a first adjacent target block immediately adjacent to the target block by confirming that the first adjacent candidate block differs from the first adjacent target block by no more than an allowed number of bytes.
2. The method of claim 1 , further comprising effectuating storage of the candidate block at least in part by providing a reference to the target block.
3. The method of claim 1 , wherein the method further comprises, in response to determining that the first adjacent candidate block matches the first adjacent target block, effectuating storage of the first adjacent candidate block by (i) providing a reference to the first adjacent target block and (ii) providing a reference to difference data computed based on a difference between the first adjacent candidate block and the first adjacent target block.
4. The method of claim 3 , wherein the method further comprises computing the difference data as a bitwise XOR (exclusive-OR) of the first adjacent candidate block with the first adjacent target block.
5. The method of claim 3 , further comprising providing a single virtual block to includes the reference to the first adjacent target block and the reference to the difference data.
6. The method of claim 1 , further comprising, in response to determining that the first adjacent candidate block matches the first adjacent target block:
determining whether a second adjacent candidate block immediately adjacent to the first adjacent candidate block matches a second adjacent target block immediately adjacent to the first adjacent target block by confirming that the second adjacent candidate block differs from the second adjacent target block by less than the predetermined amount; and
effectuating storage of the second adjacent candidate block by (i) providing a reference to the second adjacent target block and (ii) providing a reference to second difference data computed based on a difference between the second adjacent candidate block and the second adjacent target block.
7. The method of claim 1 , wherein the candidate block and the first adjacent candidate block are received together in a write request from an application in which the first adjacent candidate block precedes the candidate block.
8. The method of claim 1 , wherein the candidate block and the first adjacent candidate block are received together in a write request from an application in which the first adjacent candidate block follows the candidate block.
9. A computerized apparatus, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
access a candidate block and a set of adjacent candidate blocks adjacent to the candidate block;
in response to finding a matching entry in a database to the candidate block, access a target block referenced by the matching entry in the database and access a set of adjacent target blocks adjacent to the target block;
perform a data comparison between at least one of the set of adjacent candidate blocks and at least one of the set of adjacent target blocks; and
in response to the data comparison indicating one or more matches, effectuate storage of each matched adjacent candidate block at least in part by providing a reference to the respective matching adjacent target block,
wherein the control circuitry constructed and arranged to perform the data comparison is further constructed and arranged to determine whether a first adjacent candidate block immediately adjacent to the candidate block matches a first adjacent target block immediately adjacent to the target block by confirming that the first adjacent candidate block differs from the first adjacent target block by less than a predetermined amount, the predetermined amount being no more than an allowed number of bytes.
10. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of performing data deduplication, the method comprising:
accessing a candidate block and a set of adjacent candidate blocks adjacent to the candidate block;
in response to finding a matching entry in a database to the candidate block, accessing a target block referenced by the matching entry in the database and accessing a set of adjacent target blocks adjacent to the target block;
performing a data comparison between at least one of the set of adjacent candidate blocks and at least one of the set of adjacent target blocks; and
in response to the data comparison indicating one or more matches, effectuating storage of each matched adjacent candidate block at least in part by providing a reference to the respective matching adjacent target block,
wherein performing the data comparison includes determining whether a first adjacent candidate block immediately adjacent to the candidate block matches a first adjacent target block immediately adjacent to the target block by confirming that the first adjacent candidate block differs from the first adjacent target block by no more than an allowed number of bytes.
11. The computer program product of claim 10 , wherein the method further comprises effectuating storage of the candidate block at least in part by providing a reference to the target block.
12. The computer program product of claim 10 , wherein the method further comprises, in response to determining that the first adjacent candidate block matches the first adjacent target block, effectuating storage of the first adjacent candidate block by (i) providing a reference to the first adjacent target block and (ii) providing a reference to difference data computed based on a difference between the first adjacent candidate block and the first adjacent target block.
13. The computer program product of claim 12 , wherein the method further comprises computing the difference data as a bitwise XOR (exclusive-OR) of the first adjacent candidate block with the first adjacent target block.
14. The computer program product of claim 10 , wherein the method further comprises, in response to determining that the first adjacent candidate block matches the first adjacent target block:
determining whether a second adjacent candidate block immediately adjacent to the first adjacent candidate block matches a second adjacent target block immediately adjacent to the first adjacent target block by confirming that the second adjacent candidate block differs from the second adjacent target block by less than the predetermined amount; and
effectuating storage of the second adjacent candidate block by (i) providing a reference to the second adjacent target block and (ii) providing a reference to second difference data computed based on a difference between the second adjacent candidate block and the second adjacent target block.
15. The computer program product of claim 10 , wherein the candidate block and the first adjacent candidate block are received together in a write request from an application in which the first adjacent candidate block precedes the candidate block.
16. The computer program product of claim 10 , wherein the candidate block and the first adjacent candidate block are received together in a write request from an application in which the first adjacent candidate block follows the candidate block.