Compressing data using deduplication-like methods
A technique for performing data reduction applies deduplication principles when performing data compression, providing a form of enhanced compression. The technique obtains a chunk of data that contains multiple extents and applies deduplication actions to identify duplicate extents within the chunk. The technique marks duplicate extents in metadata. Such duplicate extents need not be compressed using conventional data compression, saving computational resources and considerable time.
1. A method of managing data storage in a computerized system, the method comprising:
receiving specified data into a cache in response to write requests;
obtaining a chunk of data from the specified data in the cache, the chunk including a plurality of extents, the plurality of extents including unique extents and duplicate extents and forming a local deduplication domain limited to the plurality of extents;
generating deduplication metadata to reflect the duplicate extents of the plurality of extents within the local deduplication domain;
computing a compressed dataset from the chunk, the compressed dataset including compressed versions of the unique extents and a dictionary that includes both (i) compression symbols used for compressing the unique extents and (ii) the deduplication metadata that reflects the duplicate extents; and
storing the compressed dataset in the computerized system.
2. The method of claim 1 , wherein the deduplication metadata identifies the duplicate extents and their locations within the chunk.
3. The method of claim 2 , wherein generating the deduplication metadata is performed when computing the compressed dataset.
4. The method of claim 1 , wherein computing the compressed dataset for the chunk is performed while or after generating the deduplication metadata.
5. The method of claim 1 , wherein computing the compressed dataset uses a compression window that has a compression size smaller than any respective size of any extent of the unique extents.
6. The method of claim 1 , further comprising restoring the chunk based at least in part on the deduplication metadata.
7. The method of claim 1 , wherein generating deduplication metadata identifies duplicate extents within the chunk and not within other chunks.
8. The method of claim 1 , further comprising:
comparing particular extents within the plurality of extents to identify the duplicate extents and corresponding locations of the duplicate extents.
9. The method of claim 1 , further comprising identifying the duplicate extents by computing hash digests for the plurality of extents and performing lookups by hash digests and determining matches between or among the hash digests.
10. The method of claim 1 , wherein each extent of the plurality of extents is any of a sector, a block, and a sub-block.
11. The method of claim 1 , wherein the plurality of extents are uniform in size.
12. The method of claim 1 , wherein the chunk has its own deduplication cache whose entries are limited to the plurality of extents, and wherein the deduplication cache is not persistently stored.
13. The method of claim 1 , wherein the computerized system processes the plurality of extents extent-by-extent in an order such that, for each current extent, the computerized system performs compression on the current extent responsive to determining that the current extent is a unique extent, and the computerized systems performs deduplication on the current extent responsive to determining that the current extent is a duplicate extent.
14. The method of claim 1 , wherein the computerized system identifies duplicate extents as extents that match a previously processed extent in the chunk of data, and wherein the computerized system identifies unique extents as extents that do not match any previously processed extent in the chunk of data.
15. A computerized system, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
receive specified data into a cache in response to write requests;
obtain a chunk of data from the specified data in the cache, the chunk including a plurality of extents, the plurality of extents including unique extents and duplicate extents and forming a local deduplication domain limited to the plurality of extents;
generate deduplication metadata to reflect the duplicate extents of the plurality of extents within the local deduplication domain;
compute a compressed dataset from the chunk, the compressed dataset including compressed versions of the unique extents and a dictionary that includes both (i) compression symbols used for compressing the unique extents and (ii) the deduplication metadata that reflects the duplicate extents; and
store the compressed dataset in the computerized system.
16. The system of claim 15 , wherein the deduplication metadata identifies the duplicate extents and their locations within the chunk.
17. The system of claim 16 , the control circuitry being further constructed and further arranged to generate the deduplication metadata when computing the compressed dataset.
18. The system of claim 17 , the control circuitry being further constructed and further arranged to store a dictionary in the compressed dataset, the dictionary including symbols used to perform compression.
19. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized system, cause the control circuitry to perform a method of managing data storage, the method comprising:
receiving specified data into a cache in response to write requests;
obtaining a chunk of data from the specified data in the cache, the chunk including a plurality of extents, the plurality of extents forming a local deduplication domain limited to the plurality of extents;
generating deduplication metadata to reflect the duplicate extents of the plurality of extents within the local deduplication domain;
computing a compressed dataset from the chunk, the compressed dataset including compressed versions of the unique extents and a dictionary that includes both (i) compression symbols used for compressing the unique extents and (ii) the deduplication metadata that reflects the duplicate extents; and
storing the compressed dataset in the computerized system.
20. The computer program product of claim 19 , wherein the deduplication metadata identifies the duplicate extents and their locations within the chunk.
21. The computer program product of claim 20 , wherein generating the deduplication metadata is performed when computing the compressed dataset.