Processing device utilizing polynomial-based signature subspace for efficient generation of deduplication estimate
An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to identify a dataset to be scanned to generate a deduplication estimate for that dataset, to designate a subset inclusion characteristic to be utilized in the scan, and for each of a plurality of pages of the dataset, to scan the page, where scanning the page includes computing a polynomial-based signature for the page, determining whether or not the polynomial-based signature satisfies the designated subset inclusion characteristic, and responsive to the polynomial-based signature satisfying the designated subset inclusion characteristic, computing a content-based signature for the page and updating a corresponding entry of a deduplication estimate table for the dataset based at least in part on the content-based signature. The processing device generates the deduplication estimate for the dataset based at least in part on contents of the deduplication estimate table.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the processing device being configured:
to identify a dataset to be scanned to generate a deduplication estimate for that dataset;
to designate a subset inclusion characteristic to be utilized in the scan;
for each of a plurality of pages of the dataset, to scan the page by:
computing a polynomial-based signature for the page;
determining whether or not the polynomial-based signature satisfies the designated subset inclusion characteristic; and
responsive to the polynomial-based signature satisfying the designated subset inclusion characteristic, computing a content-based signature for the page and updating a corresponding entry of a deduplication estimate table for the dataset based at least in part on the content-based signature; and
to generate the deduplication estimate for the dataset based at least in part on contents of the deduplication estimate table;
wherein for a first one of the pages for which the polynomial-based signature of that page satisfies the designated subset inclusion characteristic, a content-based signature is computed for that page and an update is made to the deduplication estimate table; and
wherein for a second one of the pages for which the polynomial-based signature of that page does not satisfy the designated subset inclusion characteristic, a content-based signature is not computed for that page and no update is made to the deduplication estimate table;
the designated subset inclusion characteristic thereby establishing a sampling ratio of the pages for the scan;
the designated subset inclusion characteristic specifying that application of a designated function to the polynomial-based signature produces a particular result; and
the designated function comprising a designated modulo arithmetic operation.
2. The apparatus of claim 1 wherein the processing device is implemented in one of:
a host device configured to communicate over a network with a storage system that stores the dataset; and
the storage system that stores the dataset.
3. The apparatus of claim 1 wherein the dataset comprises a set of one or more logical storage volumes of a storage system.
4. The apparatus of claim 1 wherein updating a corresponding entry of the deduplication estimate table for a given one of the pages of the dataset comprises one of the following operations (i) and (ii):
(i) responsive to a page identifier of the given page not already being present in the deduplication estimate table, inserting the page identifier into the deduplication estimate table and setting an associated counter to an initial value; and
(ii) responsive to the page identifier already being present in the deduplication estimate table, incrementing its associated counter.
5. The apparatus of claim 1 wherein the deduplication estimate table comprises a plurality of entries for respective ones of the pages and wherein each of the entries is configured to include a page identifier that comprises less than an entire content-based signature of its corresponding page.
6. The apparatus of claim 5 wherein the processing device is configured to adjust at least one of an amount of computational resources and an amount of memory resources to be utilized in the scan of the dataset at least in part by altering a length of the page identifier.
7. The apparatus of claim 5 wherein the processing device is configured to select a length of the page identifier in order to achieve a rate of false positives in the deduplication estimate that is less than a specified maximum rate of false positives.
8. The apparatus of claim 1 wherein the polynomial-based signature comprises an n-bit cyclic redundancy check (CRC) value.
9. The apparatus of claim 8 wherein the n-bit CRC value comprises a 32-bit CRC value.
10. The apparatus of claim 1 wherein the designated subset inclusion characteristic specifies that performing the designated modulo arithmetic operation on the polynomial-based signature produces a particular value.
11. The apparatus of claim 1 wherein generating the deduplication estimate for the dataset based at least in part on contents of the deduplication estimate table further comprises:
computing a partial deduplication estimate based at least in part on values of counters associated with respective entries of the deduplication estimate table; and
scaling the partial deduplication estimate to obtain the deduplication estimate for the dataset.
12. The apparatus of claim 1 wherein the processing device is configured to adjust one or more characteristics of a storage configuration of the dataset based at least in part on the deduplication estimate generated for the dataset.
13. The apparatus of claim 1 wherein the processing device is configured:
to generate one or more additional deduplication estimates for respective ones of one or more additional datasets; and
to select a particular one of the datasets for deduplication based at least in part on their respective deduplication estimates.
14. A method comprising:
identifying a dataset to be scanned to generate a deduplication estimate for that dataset;
designating a subset inclusion characteristic to be utilized in the scan;
for each of a plurality of pages of the dataset, scanning the page by:
computing a polynomial-based signature for the page;
determining whether or not the polynomial-based signature satisfies the designated subset inclusion characteristic; and
responsive to the polynomial-based signature satisfying the designated subset inclusion characteristic, computing a content-based signature for the page and updating a corresponding entry of a deduplication estimate table for the dataset based at least in part on the content-based signature; and
generating the deduplication estimate for the dataset based at least in part on contents of the deduplication estimate table;
wherein for a first one of the pages for which the polynomial-based signature of that page satisfies the designated subset inclusion characteristic, a content-based signature is computed for that page and an update is made to the deduplication estimate table; and
wherein for a second one of the pages for which the polynomial-based signature of that page does not satisfy the designated subset inclusion characteristic, a content-based signature is not computed for that page and no update is made to the deduplication estimate table;
the designated subset inclusion characteristic thereby establishing a sampling ratio of the pages for the scan;
the designated subset inclusion characteristic specifying that application of a designated function to the polynomial-based signature produces a particular result; and
the designated function comprising a designated modulo arithmetic operation;
wherein the method is implemented by at least one processing device comprising a processor coupled to a memory.
15. The method of claim 14 wherein the polynomial-based signature comprises an n-bit cyclic redundancy check (CRC) value.
16. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to identify a dataset to be scanned to generate a deduplication estimate for that dataset;
to designate a subset inclusion characteristic to be utilized in the scan;
for each of a plurality of pages of the dataset, to scan the page by:
computing a polynomial-based signature for the page;
determining whether or not the polynomial-based signature satisfies the designated subset inclusion characteristic; and
responsive to the polynomial-based signature satisfying the designated subset inclusion characteristic, computing a content-based signature for the page and updating a corresponding entry of a deduplication estimate table for the dataset based at least in part on the content-based signature; and
to generate the deduplication estimate for the dataset based at least in part on contents of the deduplication estimate table;
wherein for a first one of the pages for which the polynomial-based signature of that page satisfies the designated subset inclusion characteristic, a content-based signature is computed for that page and an update is made to the deduplication estimate table; and
wherein for a second one of the pages for which the polynomial-based signature of that page does not satisfy the designated subset inclusion characteristic, a content-based signature is not computed for that page and no update is made to the deduplication estimate table;
the designated subset inclusion characteristic thereby establishing a sampling ratio of the pages for the scan;
the designated subset inclusion characteristic specifying that application of a designated function to the polynomial-based signature produces a particular result; and
the designated function comprising a designated modulo arithmetic operation.
17. The computer program product of claim 16 wherein the polynomial-based signature comprises an n-bit cyclic redundancy check (CRC) value.
18. The computer program product of claim 16 wherein the designated subset inclusion characteristic specifies that performing the designated modulo arithmetic operation on the polynomial-based signature produces a particular value.
19. The computer program product of claim 16 wherein generating the deduplication estimate for the dataset based at least in part on contents of the deduplication estimate table further comprises:
computing a partial deduplication estimate based at least in part on values of counters associated with respective entries of the deduplication estimate table; and
scaling the partial deduplication estimate to obtain the deduplication estimate for the dataset.
20. The method of claim 14 wherein the designated subset inclusion characteristic specifies that performing the designated modulo arithmetic operation on the polynomial-based signature produces a particular value.