Performing compression and deduplication at different granularities
A technique for promoting storage efficiency in a data storage system decouples data-set sizes used for compression from those used for deduplication, by performing deduplication at extent-level granularity but performing compression on aggregates that contain multiple extents. Compression and deduplication can each work on data-set sizes that are more optimal for the respective features, thereby avoiding the need to settle on a compromise that is not particularly well-suited to either of them.
1. A method of managing data storage in a computerized system that stores data within blocks, the method comprising:
grouping blocks into aggregates, each aggregate including at least two blocks, the blocks having a common block size;
performing data deduplication at block-level granularity, including matching new blocks to already-stored blocks in the computerized system; and
performing data compression at aggregate-level granularity by compressing all blocks within each aggregate together as a single data set,
wherein the method further comprises providing access to the blocks using mapping metadata that includes (i) pointer arrays and (ii) virtualization structures, each pointer array including a respective set of pointers, each pointer pointing to a respective virtualization structure that points in turn to a respective block.
2. The method of claim 1 wherein performing data deduplication includes providing a digest database that relates digests of blocks to respective locations of those blocks in the computerized system, and wherein matching new blocks to already-stored blocks includes computing a digest of a new block and performing a lookup into the digest database for the computed digest.
3. The method of claim 1 , wherein a virtualization structure that points to a particular block includes a pointer to a particular aggregate that includes the particular block, the particular block having been compressed as part of a data set that includes all blocks grouped together within the particular aggregate.
4. The method of claim 3 , wherein the virtualization structure that points to the particular block includes tracking metadata that identifies the particular block within the particular aggregate.
5. The method of claim 4 , wherein the tracking metadata includes an index and a length, the index identifying a position of the particular block among all of the blocks within the particular aggregate, the length indicating an uncompressed size of the particular aggregate.
6. The method of claim 5 , wherein the uncompressed size of the particular aggregate in the tracking metadata is provided as an integer number of blocks.
7. The method of claim 4 , further comprising enforcing an upper limit on a number of blocks that are grouped together in a single aggregate.
8. The method of claim 7 , wherein the upper limit corresponds to 32 kB (kilobytes) per aggregate.
9. The method of claim 7 , wherein the upper limit is adjustable dynamically based upon a load on the computerized system.
10. The method of claim 9 , wherein the upper limit is further adjustable dynamically based upon at least one of the following:
a compression algorithm used to compress aggregates; and
whether blocks stored within an aggregate correspond to logically contiguous data.
11. The method of claim 4 , wherein grouping blocks into aggregates includes placing a set of blocks having contiguous logical offsets into contiguous positions within an aggregate.
12. The method of claim 4 , further comprising responding to a read request directed to a specified logical address of the particular block by:
accessing a pointer array that maps the specified logical address;
following a pointer in the accessed pointer array to the virtualization structure provided for the particular block;
reading the particular aggregate pointed to by the virtualization structure;
decompressing the particular aggregate;
applying the tracking metadata within the virtualization structure to locate the particular block within the particular aggregate; and
returning the particular block to satisfy the read request.
13. The method of claim 4 , wherein the digest database stores the respective locations of the blocks by identifying virtualization structures that point to the respective blocks in the computerized system.
14. The method of claim 13 , further comprising providing a reference count for the virtualization structure that points to the particular block, the reference count tracking a number of pointers in the pointer arrays that point to the virtualization structure based at least in part on having performed deduplication to the particular block.
15. A computerized system, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
group blocks into aggregates, each aggregate including at least two blocks, the blocks having a common block size;
perform data deduplication at block-level granularity, including matching new blocks to already-stored blocks in the computerized system; and
perform data compression at aggregate-level granularity by compressing all blocks within each aggregate together as a single data set,
wherein the control circuitry is further constructed and arranged to provide access to the blocks using mapping metadata that includes (i) pointer arrays and (ii) virtualization structures, each pointer array including a respective set of pointers, each pointer pointing to a respective virtualization structure that points in turn to a respective block.
16. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized system, cause the control circuitry to perform a method of managing data storage, the method comprising:
grouping blocks into aggregates, each aggregate including at least two blocks, the blocks having a common block size;
performing data deduplication at block-level granularity, including matching new blocks to already-stored blocks in the computerized system; and
performing data compression at aggregate-level granularity by compressing all blocks within each aggregate together as a single data set,
wherein the method further comprises providing access to the blocks using mapping metadata that includes (i) pointer arrays and (ii) virtualization structures, each pointer array including a respective set of pointers, each pointer pointing to a respective virtualization structure that points in turn to a respective block.
17. The computer program product of claim 16 ,
wherein a virtualization structure that points to a particular block includes a pointer to a particular aggregate that includes the particular block, the particular block having been compressed as part of a data set that includes all blocks grouped together within the particular aggregate.
18. The computer program product of claim 17 , wherein the virtualization structure that points to the particular block includes tracking metadata that identifies the particular block within the particular aggregate.
19. The computer program product of claim 16 , further comprising enforcing an upper limit on a number of blocks that are grouped together in a single aggregate.