Secure data deduplication with smaller hash values
Techniques for providing secure deduplication in a data storage system using smaller hash values. The techniques employ a keyed hash function to generate keyed hash values for data blocks received at the data storage system. The keyed hash function can generate keyed hash values having an acceptable probability of accidental collision, in which each keyed hash value requires a reduced number of bits to represent them. By representing each keyed hash value with a number of bits less than the number required for a cryptographic hash value, the impact that an index table in main memory has on the amount of free memory space available can be reduced, while still providing an acceptable probability of accidental collision. The keyed hash function can be implemented as a keyed universal hash function, which can reduce the number of processor cycles required to generate a keyed hash value for each received data block.
1. A method of providing secure data deduplication with smaller hash values in a data storage system, the data storage system including a storage processor and a memory, the method comprising:
randomly generating a plurality of noncryptographic keys for a plurality of deduplication domains, respectively, the plurality of noncryptographic keys including a noncryptographic key for a respective deduplication domain from among the plurality of deduplication domains;
receiving, at the storage processor, data for storage in one or more storage resources of the respective deduplication domain;
segmenting the data into a plurality of data segments;
for at least a respective data segment from among the plurality of data segments, inputting only the respective data segment and the noncryptographic key for the respective deduplication domain into a universal hash function based on a Galois Message Authentication Code (GMAC) to obtain a keyed hash value for the respective data segment that has an acceptable probability of accidental collision for secure data deduplication;
maintaining the obtained keyed hash value for the respective data segment in an index table for the respective deduplication domain; and
performing a data deduplication operation on the respective data segment using the keyed hash value for the respective data segment.
2. The method of claim 1 further comprising:
storing the index table in the memory of the data storage system.
3. The method of claim 1 wherein the storage resources of the deduplication domain are located in a storage pool, and wherein the method further comprises:
storing the index table in the same storage pool as the storage resources of the deduplication domain.
4. The method of claim 1 wherein the performing of the data deduplication operation includes:
comparing the keyed hash value for the respective data segment with the respective keyed hash values maintained in the index table; and
determining whether the keyed hash value for the respective data segment matches any of the respective keyed hash values maintained in the index table.
5. The method of claim 4 further comprising:
having determined that the keyed hash value for the respective data segment matches one of the respective keyed hash values maintained in the index table, (i) generating metadata for the respective data segment, the metadata including the keyed hash value of the respective data segment, a pointer to a storage location of a data segment having the matching keyed hash value, and a reference to an original location of the respective data segment in the received data, and (ii) removing the respective data segment from the data storage system.
6. The method of claim 4 further comprising:
having determined that the keyed hash value for the respective data segment does not match any of the respective keyed hash values maintained in the index table, (i) storing the respective data segment in association with one or more other data segments from the received data at a storage location in one of the storage resources of the deduplication domain, and (ii) incorporating the keyed hash value of the respective data segment and a pointer that maps the keyed hash value of the respective data segment to the storage location of the respective data segment into a new entry for the index table.
7. The method of claim 1 further comprising:
accessing, over a network, the noncryptographic key for the respective deduplication domain of the one or more storage resources from a secure key server computer.
8. The method of claim 7 wherein the accessing of the noncryptographic key for the respective deduplication domain includes issuing, by the storage processor, a key request to the secure key server computer to obtain the noncryptographic key over the network.
9. A system for providing secure data deduplication with smaller hash values, comprising:
a plurality of storage resources of a respective deduplication domain from among a plurality of deduplication domains;
a memory; and
a storage processor operative to execute instructions out of the memory: to randomly generate a plurality of noncryptographic keys for a plurality of deduplication domains, respectively, the plurality of noncryptographic keys including a noncryptographic key for the respective deduplication domain;
to receive data for storage in one or more of the plurality of storage resources of the respective deduplication domain; to segment the data into a plurality of data segments;
for at least a respective data segment from among the plurality of data segments, to input only the respective data segment and the noncryptographic key for the respective deduplication domain into a universal hash function based on a Galois Message Authentication Code (GMAC) to obtain a keyed hash value for the respective data segment that has an acceptable probability of accidental collision for secure data deduplication;
to maintain the obtained keyed hash value for the respective data segment in an index table for the respective deduplication domain; and
to perform a data deduplication operation on the respective data segment using the keyed hash value for the respective data segment.
10. The system of claim 9 wherein the storage processor is further operative to execute the instructions out of the memory to store the index table in the memory.
11. The system of claim 9 wherein the plurality of storage resources of the deduplication domain is located in a storage pool, and wherein the storage processor is further operative to execute the instructions out of the memory to store the index table in the same storage pool as the plurality of storage resources of the deduplication domain.
12. A computer program product having a non-transitory computer readable medium that stores a set of instructions to provide secure data deduplication with smaller hash values in a data storage system, the set of instructions, when carried out by computerized circuitry, causing the computerized circuitry to perform a method of:
randomly generating a plurality of noncryptographic keys for a plurality of deduplication domains, respectively, the plurality of noncryptographic keys including a noncryptographic key for a respective deduplication domain from among the plurality of deduplication domains;
receiving, at a storage processor, data for storage in one or more storage resources of the respective deduplication domain; segmenting the data into a plurality of data segments;
for at least a respective data segment from among the plurality of data segments, inputting only the respective data segment and the noncryptographic key for the respective deduplication domain into a universal hash function based on a Galois Message Authentication Code (GMAC) to obtain a keyed hash value for the respective data segment that has an acceptable probability of accidental collision for secure data deduplication;
maintaining the obtained keyed hash value for the respective data segment in an index table for the respective deduplication domain; and
performing a data deduplication operation on the respective data segment using the keyed hash value for the respective data segment.
13. The method of claim 1 wherein the keyed hash value for the respective data segment is represented by the number of bits selected from the group consisting of 32 , 64 , and 96 .