Deduplication of large block aggregates using representative block digests
A method of performing deduplication includes (1) receiving a write command that specifies a set of data, the set of data including multiple blocks of data, (2) hashing a subset of the set of data, yielding a representative digest of the set of data, and (3) performing deduplication on the set of data based at least in part on matching the representative digest to a digest already stored in a database which relates digests to locations of data from which the digests were produced. An apparatus, system, and computer program product for performing a similar method are also provided.
1. A method comprising:
receiving a write command that specifies a set of data, the set of data including multiple blocks of data;
hashing a subset of the set of data, yielding a representative digest of the set of data;
determining a size of the set of data; and
performing deduplication on the set of data based at least in part on matching the representative digest to a digest already stored in a database which relates digests to locations of data from which the digests were produced, databases assigned to respective sizes by choosing the database that is assigned to the determined size.
2. The method of claim 1 wherein determining the size of the set of data is performed after the multiple blocks have been stored in a write cache commingled with other blocks of data that are part of other write commands.
3. The method of claim 2 wherein determining the size of the set of data includes searching the write cache for blocks having associated metadata indicating that they:
(a) were received simultaneously and
(b) are to be written to consecutive addresses of a same logical disk.
4. The method of claim 3 wherein searching the write cache further includes searching for a group of blocks having exactly an integer-power-of-2 number of blocks within a range from 2 through 32, inclusive.
5. The method of claim 1 wherein the set of databases include at least a first database and a second database respectively assigned to sizes of 16 kilobytes and 64 kilobytes.
6. The method of claim 1 wherein the set of databases includes at least 2 databases respectively assigned to an integer-power-of-2 byte size within a range of [2 12 , 2 17 ] bytes.
7. The method of claim 1 wherein performing deduplication on the set of data is further based on, in response to matching the representative digest to the digest already stored in the database, successfully comparing the set of data with data identified by the database in connection with the representative digest.
8. A method comprising:
receiving a write command that specifies a set of data, the set of data including multiple blocks of data;
hashing a subset of the set of data, yielding a representative digest of the set of data, wherein the subset includes a first block at a beginning of the set of data;
hashing a last block at a tail end of the set of data, yielding another representative digest of the set of data; and
performing deduplication on the set of data based at least in part on matching the representative digest to a digest already stored in a database which related digests to locations of data from which the digests were produced;
wherein matching the representative digest to the digest already stored in the database includes finding an entry of the database keyed by the representative digest; and
wherein performing deduplication on the set of data is further based on determining, in response to finding the entry, that the entry also includes the other representative digest.
9. The method of claim 8 wherein performing deduplication on the set of data is further based on, in response to determining that the entry also includes the other representative digest, successfully comparing the set of data with another set of data identified by the entry.
10. An apparatus comprising processing circuitry coupled to memory configured to:
receive a write command that specifies a set of data, the set of data including multiple blocks of data;
hash a subset of the set of data, yielding a representative digest of the set of data;
determine a size of the set of data; and
perform deduplication on the set of data based at least in part on matching the representative digest to a digest already stored in a database which relates digests to locations of data from which the digests were produced, wherein performing deduplication includes selecting the database from a set of databases assigned to respective sizes by choosing the database that is assigned to the determined size.
11. The apparatus of claim 10 , wherein the subset includes a first block at a beginning of the set of data.
12. A computer program product comprising a non-transitory computer-readable storage medium storing instructions, which, when executed by processing circuitry, cause a computing device to:
receive a write command that specifies a set of data, the set of data including multiple blocks of data;
hash a subset of the set of data, yielding a representative digest of the set of data;
determine a size of the set of data; and
perform deduplication on the set of data based at least in part on matching the representative digest to a digest already stored in a database which relates digests to locations of data from which the digests were produced, wherein performed deduplication includes selecting the database from a set of databases assigned to respective sizes by choosing the database that is assigned to the determined size.
13. The computer program product of claim 12 , wherein the subset includes a first block at a beginning of the set of data.