Method and apparatus for accelerating deduplication processing
A deduplication engine maintains a hash table containing hash values of tracks of data stored on managed drives of a storage system. The deduplication engine keeps track of how frequently the tracks are accessed by the deduplication engine using an exponential moving average for each track. Target tracks which are frequently accessed by the deduplication engine are cached in local memory, so that required byte-by-byte comparisons between the target track and write data may be performed locally rather than requiring the target track to be read from managed drives. The deduplication engine implements a Least Recently Used (LRU) cache data structure in local memory to manage locally cached tracks of data. If a track is to be removed from local memory, a final validation of the target track is implemented on the version stored in managed resources before evicting the track from the LRU cache.
1. A non-transitory tangible computer readable storage medium having stored thereon a computer program for accelerating deduplication processing, the computer program including a set of instructions which, when executed by a computer, cause the computer to perform a method comprising the steps of:
maintaining a hash table containing a plurality of entries, each entry identifying a track of data stored on managed drives, a hash value computed from data stored at the identified track, an exponential moving average indicating a number of times a deduplication engine has accessed the data stored at the identified track in a recent time interval, and a flag indicating whether the data stored at the identified track has been cached in local memory or in global memory;
receiving a write operation containing write data;
calculating a hash value of the write data;
using the hash value of the write data to determine whether any entry of the hash table has the same hash value;
if the hash value of the write data matches a respective one of the entries of the hash table, reading a respective flag for the entry to determine if data stored at a respective track of managed storage resources (target data) has been stored in local memory or in global memory;
if the respective flag indicates that the target data has been stored in local memory or in global memory, performing a byte-to-byte comparison between the write data and the target data based on the copy of the target data in local memory or in global memory; and
if the respective flag indicates that the target data has not been cached in local memory or in global memory, performing the byte-to-byte comparison between the write data and the target data by reading the target data from the respective track of managed storage resources.
2. The non-transitory tangible computer readable storage medium of claim 1 , the method further comprising incrementing a deduplication counter for the target data in the hash table.
3. The non-transitory tangible computer readable storage medium of claim 1 , the method further comprising calculating an exponential moving average number of hits on the target data in a preceding time interval.
4. The non-transitory tangible computer readable storage medium of claim 3 , the method further comprising if the flag indicates that the target data has not been cached in local memory, and the exponential moving average number of hits on the target data exceeds a first threshold, copying the target data to local memory.
5. The non-transitory tangible computer readable storage medium of claim 3 , the method further comprising if the flag indicates that the target data has been cached in local memory, and the exponential moving average number of hits on the target data exceeds a second threshold, copying the target data to global memory.
6. The non-transitory tangible computer readable storage medium of claim 1 , the method further comprising implementing a least recently used cache data structure in local memory.
7. The non-transitory tangible computer readable storage medium of claim 6 , the method further comprising copying the target data to local memory and evicting a least recently used track of data from the least recently used cache data structure.
8. The non-transitory tangible computer readable storage medium of claim 7 , wherein evicting the least recently used track of data from the least recently used cache data structure comprises reading a corresponding track of data from managed storage resources, performing a byte-to-byte comparison between the least recently used track of data and corresponding track of data from managed resources, and verifying a linkage of the corresponding track of data in managed resources.
9. The non-transitory tangible computer readable storage medium of claim 8 , the method further comprising removing the least recently used track of data from the cache and resetting a corresponding flag for the least recently used track of data in the hash table.
10. The non-transitory tangible computer readable storage medium of claim 1 , the method further comprising:
if the write operation comprises changing the content of a first track of data that is currently stored in managed storage resources, performing the steps of:
removing a respective first hash table entry for the first track of data from the hash table;
reading a respective first flag of the first hash table entry to determine if the first track of data has been stored in local memory or in global memory; and
if the respective first flag of the first hash table entry indicates that the first track of data has been stored in local memory or in global memory, removing the first track of data from local memory or global memory.
11. The non-transitory tangible computer readable storage medium of claim 1 , the method further comprising maintaining a set of deduplication cache policies associated with criteria for storing tracks of data in local memory or global memory.
12. The non-transitory tangible computer readable storage medium of claim 11 , wherein a first of the deduplication cache policies specifies an amount of local memory to be used for storing tracks of data for deduplication processing.
13. The non-transitory tangible computer readable storage medium of claim 12 , wherein a second of the deduplication cache policies specifies an exponential moving average threshold for selecting tracks of data to be stored in the local memory or global memory.
14. The non-transitory tangible computer readable storage medium of claim 13 , wherein a third of the deduplication cache policies specifies a processes to be followed in connection with removing tracks of data from local memory or global memory.