Extending similarity-based deduplication to adjacent data
A technique of performing data reduction includes, upon detecting a match between similarity hashes of a candidate dataset and a target dataset, evaluating an adjacent candidate dataset and an adjacent target dataset for similarity with each other and, in response to confirming such similarity, performing data reduction of the adjacent candidate dataset with reference to the adjacent target dataset.
1. A method of performing data reduction, comprising:
receiving a sequence of datasets to be written in a data storage system, the sequence of datasets including a candidate dataset and an adjacent candidate dataset;
upon detecting a match between similarity hashes of the candidate dataset and a target dataset, performing a similarity assessment between the adjacent candidate dataset and an adjacent target dataset adjacent to the target dataset; and
in response to the similarity assessment determining that the adjacent candidate dataset and the adjacent target dataset are similar to at least a predetermined degree, performing a data reduction operation on the adjacent candidate dataset with reference to the adjacent target dataset,
wherein performing the similarity assessment includes:
accessing P hash values calculated from the adjacent candidate dataset and P hash values calculated from the adjacent target dataset;
selecting N of the P hash values, N<P and being less than one-tenth of P, of the adjacent candidate block based on a selection rule;
selecting N of the P hash values of the adjacent target block based on the same selection rule; and
determining that the adjacent candidate block is similar to the adjacent target block based at least in part on a number of matches between the N selected hash values of the adjacent candidate block and the N selected hash values of the adjacent target block.
2. The method of claim 1 , wherein performing the data reduction operation includes performing a dictionary-based compression of the adjacent candidate dataset based on the adjacent target dataset.
3. The method of claim 2 , further comprising:
performing a second similarity assessment between a second-adjacent candidate dataset adjacent to the adjacent candidate dataset and a second-adjacent target dataset adjacent to the adjacent target dataset; and
in response to the second similarity assessment determining that the second-adjacent candidate dataset and the second-adjacent target dataset are not similar to at least the predetermined degree, performing a data reduction operation on the second-adjacent candidate dataset without reference to the second-adjacent target dataset.
4. The method of claim 3 , wherein performing the data reduction operation on the second-adjacent candidate dataset includes performing a self-compression of the second-adjacent candidate dataset.
5. The method of claim 1 , wherein the N selected hash values of the adjacent candidate dataset are pre-computed by generating a similarity hash of the adjacent candidate dataset.
6. The method of claim 1 , wherein selecting N of the P hash values of the adjacent candidate block based on the selection rule includes selecting one of (i) the N largest hash values of the P hash values or (ii) the N smallest hash values of the P hash values.
7. The method of claim 6 , wherein each of the P hash values is calculated from a region that is at least 4 bytes long, and wherein the number N is at least 10.
8. A computerized apparatus, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
receive a sequence of datasets to be written in a data storage system, the sequence of datasets including a candidate dataset and an adjacent candidate dataset;
upon detection of a match between similarity hashes of the candidate dataset and a target dataset, perform a similarity assessment between the adjacent candidate dataset and an adjacent target dataset adjacent to the target dataset; and
in response to a determination by the similarity assessment that the adjacent candidate dataset and the adjacent target dataset are similar to at least a predetermined degree, perform a data reduction operation on the adjacent candidate dataset with reference to the adjacent target dataset,
wherein the control circuitry constructed and arranged to perform the similarity assessment is further constructed and arranged to:
access P hash values calculated from the adjacent candidate dataset and P hash values calculated from the adjacent target dataset;
select N of the P hash values, N<P and being less than one-tenth of P, of the adjacent candidate block based on a selection rule;
select N of the P hash values of the adjacent target block based on the same selection rule; and
determine that the adjacent candidate block is similar to the adjacent target block based at least in part on a number of matches between the N selected hash values of the adjacent candidate block and the N selected hash values of the adjacent target block.
9. The computerized apparatus of claim 8 , wherein the control circuitry constructed and arranged to perform the data reduction operation is further constructed and arranged to perform a dictionary-based compression of the adjacent candidate dataset based on the adjacent target dataset.
10. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of performing data reduction, the method comprising:
receiving a sequence of datasets to be written in a data storage system, the sequence of datasets including a candidate dataset and an adjacent candidate dataset;
upon detecting a match between similarity hashes of the candidate dataset and a target dataset, performing a similarity assessment between the adjacent candidate dataset and an adjacent target dataset adjacent to the target dataset; and
in response to the similarity assessment determining that the adjacent candidate dataset and the adjacent target dataset are similar to at least a predetermined degree, performing a data reduction operation on the adjacent candidate dataset with reference to the adjacent target dataset,
wherein performing the similarity assessment includes:
accessing P hash values calculated from the adjacent candidate dataset and P hash values calculated from the adjacent target dataset;
selecting N of the P hash values, N<P and being less than one-tenth of P, of the adjacent candidate block based on a selection rule;
selecting N of the P hash values of the adjacent target block based on the same selection rule; and
determining that the adjacent candidate block is similar to the adjacent target block based at least in part on a number of matches between the N selected hash values of the adjacent candidate block and the N selected hash values of the adjacent target block.
11. The computer program product of claim 10 , wherein the method further comprises:
performing a second similarity assessment between a second-adjacent candidate dataset adjacent to the adjacent candidate dataset and a second-adjacent target dataset adjacent to the adjacent target dataset; and
in response to the second similarity assessment determining that the second-adjacent candidate dataset and the second-adjacent target dataset are not similar to at least the predetermined degree, performing a data reduction operation on the second-adjacent candidate dataset without reference to the second-adjacent target dataset.
12. The computer program product of claim 11 , wherein performing the data reduction operation on the second-adjacent candidate dataset includes performing a self-compression of the second-adjacent candidate dataset.
13. The computer program product of claim 10 , wherein the N selected hash values of the adjacent candidate dataset are pre-computed by generating a similarity hash of the adjacent candidate dataset.
14. The computer program product of claim 10 , wherein selecting N of the P hash values of the adjacent candidate block based on the selection rule includes selecting one of (i) the N largest hash values of the P hash values or (ii) the N smallest hash values of the P hash values.
15. The computer program product of claim 14 , wherein each of the P hash values is calculated from a region that is at least 4 bytes long, and wherein the number N is at least 10.
16. The method of claim 1 , wherein P is at least 100 times greater than N.
17. The method of claim 1 , wherein the P hash values calculated from the adjacent candidate dataset are each calculated from a respective set of M consecutive bytes of the adjacent candidate dataset, and wherein each set of M consecutive bytes has at least one byte in common with at least one immediately adjacent set of M consecutive bytes.