Deduplicating data at sub-block granularity
A technique for performing data deduplication operates at sub-block granularity by searching a deduplication database for a match between a candidate sub-block of a candidate block and a target sub-block of a previously-stored target block. When a match is found, the technique identifies a duplicate range shared between the candidate block and the target block and effects persistent storage of the duplicate range by configuring mapping metadata of the candidate block so that it points to the duplicate range in the target block.
1. A method of performing data deduplication, comprising:
providing a deduplication database in a data storage system, the deduplication database configured to store multiple entries that associate digest values computed from respective sub-blocks of data blocks with references to locations in the data storage system where data blocks containing the respective sub-blocks can be found;
implementing a policy for adding entries to the deduplication database, the policy specifying creation of new entries for first sub-blocks and last sub-blocks of data blocks but not for intermediate sub-blocks of those data blocks; and
processing a new data block for deduplication by (i) identifying multiple sub-blocks of the new data block, (ii) computing respective new digest values from the sub-blocks of the new data block, (iii) matching one of the new digest values to an entry in the deduplication database, and (iv) storing the new data block at least in part by reference to a target data block whose location is referenced by the matching entry.
2. The method of claim 1 , wherein the sub-block whose digest value matches the matching entry in the deduplication database is neither a first sub-block nor a last sub-block of the new data block.
3. The method of claim 2 , wherein the new data block contains a shifted version of a set of data stored in the target data block, the set of data having a length longer than that of any of the sub-blocks of the new data block.
4. The method of claim 2 , wherein providing the deduplication database includes operating the deduplication database in memory of the data storage system, and wherein implementing the policy for adding entries to the deduplication database consumes less memory than would be consumed if new entries were created for all sub-blocks of data blocks.
5. The method of claim 2 , wherein storing the new data block at least in part by reference to the target data block includes (i) storing a first portion of the new data block by reference to the target data block and (ii) storing a second portion of the new data block in newly-allocated storage space.
6. The method of claim 2 , wherein the sub-blocks of the new data block are uniformly sized.
7. The method of claim 6 , wherein the sub-blocks of the new data block are 512 bytes long, wherein the target data block includes a matching sub-block that matches the matched sub-block from which said one of the new digest entries is computed, and wherein a position of the matching sub-block within the target data block differs from a position of the matched sub-block within the new data block by an integer multiple of 512 bytes.
8. The method of claim 1 , wherein the sub-blocks of the new data block are uniformly sized, wherein the target data block includes a matching sub-block that matches the matched sub-block from which said one of the new digest entries is computed, and wherein a position of the matching sub-block within the target data block differs from a position of the matched sub-block within the new data block by an integer number of sub-blocks.
9. A data storage system, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
provide a deduplication database, the deduplication database configured to store multiple entries that associate digest values computed from respective sub-blocks of data blocks with references to locations in the data storage system where data blocks containing the respective sub-blocks can be found;
implement a policy for adding entries to the deduplication database, the policy specifying creation of new entries for first sub-blocks and last sub-blocks of data blocks but not for intermediate sub-blocks of those data blocks; and
process a new data block for deduplication by (i) identifying multiple sub-blocks of the new data block, (ii) computing respective new digest values from the sub-blocks of the new data block, (iii) matching one of the new digest values to an entry in the deduplication database, and (iv) storing the new data block at least in part by reference to a target data block whose location is referenced by the matching entry.
10. The data storage system of claim 9 , wherein the sub-block whose digest value matches the matching entry in the deduplication database is neither a first sub-block nor a last sub-block of the new data block.
11. The data storage system of claim 10 , wherein the new data block contains a shifted version of a set of data stored in the target data block, the set of data having a length longer than that of any of the sub-blocks of the new data block.
12. The data storage system of claim 10 , wherein the control circuitry constructed and arranged to provide the deduplication database is further constructed and arranged to operate the deduplication database in memory of the data storage system, and wherein the control circuitry constructed and arranged to implement the policy for adding entries to the deduplication database is further constructed and arranged to consume less memory than would be consumed if new entries were created for all sub-blocks of data blocks.
13. The data storage system of claim 10 , wherein the control circuitry constructed and arranged to store the new data block at least in part by reference to the target data block is further constructed and arranged to (i) store a first portion of the new data block by reference to the target data block and (ii) store a second portion of the new data block in newly-allocated storage space.
14. The data storage system of claim 9 , wherein the sub-blocks of the new data block are 512 bytes long, wherein the target data block includes a matching sub-block that matches the matched sub-block from which said one of the new digest entries is computed, and wherein a position of the matching sub-block within the target data block differs from a position of the matched sub-block within the new data block by an integer multiple of 512 bytes.
15. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a data storage system, cause the control circuitry to perform a method of performing data deduplication, the method comprising:
providing a deduplication database, the deduplication database configured to store multiple entries that associate digest values computed from respective sub-blocks of data blocks with references to locations in the data storage system where data blocks containing the respective sub-blocks can be found;
implementing a policy for adding entries to the deduplication database, the policy specifying creation of new entries for first sub-blocks and last sub-blocks of data blocks but not for intermediate sub-blocks of those data blocks; and
processing a new data block for deduplication by (i) identifying multiple sub-blocks of the new data block, (ii) computing respective new digest values from the sub-blocks of the new data block, (iii) matching one of the new digest values to an entry in the deduplication database, and (iv) storing the new data block at least in part by reference to a target data block whose location is referenced by the matching entry.
16. The computer program product of claim 15 , wherein the sub-block whose digest value matches the matching entry in the deduplication database is neither a first sub-block nor a last sub-block of the new data block.
17. The computer program product of claim 16 , wherein the new data block contains a shifted version of a set of data stored in the target data block, the set of data having a length longer than that of any of the sub-blocks of the new data block.
18. The computer program product of claim 16 , wherein providing the deduplication database includes operating the deduplication database in memory of the data storage system, and wherein implementing the policy for adding entries to the deduplication database consumes less memory than would be consumed if new entries were created for all sub-blocks of data blocks.
19. The computer program product of claim 16 , wherein storing the new data block at least in part by reference to the target data block includes (i) storing a first portion of the new data block by reference to the target data block and (ii) storing a second portion of the new data block in newly-allocated storage space.
20. The computer program product of claim 16 , wherein the sub-blocks of the new data block are uniformly sized.
21. The computer program product of claim 20 , wherein the sub-blocks of the new data block are 512 bytes long, wherein the target data block includes a matching sub-block that matches the matched sub-block from which said one of the new digest entries is computed, and wherein a position of the matching sub-block within the target data block differs from a position of the matched sub-block within the new data block by an integer multiple of 512 bytes.