Combined Read/Write Cache for Deduplicated Metadata Service
In a method of processing data similarity groups, a combined read/write cache comprises an in-memory data structure having a fixed allocation of physical memory that includes a write portion comprising memory allocated to write entries and a read portion comprising memory allocated to read entries. Similarity group entries are written into the cache using keys based upon an identifier of a similarity group, and similarity group entries are read from the cache using keys based upon both similarity group and subgroup identifiers. The sizes of memory allocated to the write portion and to the read portion are dynamically varied within the fixed memory allocation based upon demand while maintaining the fixed memory allocation constant.
1 . A method of processing similarity groups for deduplication and restoration in a combined read/write cache comprising an in-memory data structure having a fixed allocation of physical memory that includes a write portion allocated to write entries and a read portion allocated to read entries, the method comprising:
entering similarity group entries into the cache;
accessing said entries in said cache by querying the cache with a key based upon an identifier of a similarity group; and
dynamically varying the sizes of memory allocated to said write portion and to said read portion based upon demand while maintaining said fixed memory allocation constant.
2 . The method of claim 1 , wherein said accessing said entries in said cache comprises accessing write entries for deduplication with a write key formed to include an identifier of a similarity group, and accessing read entries for restoration by querying the cache with a read key formed with both an identifier and a subgroup identifier of a similarity group.
3 . The method of claim 1 further comprising setting high and low cache utilization thresholds, and evicting least recently used entries from the cache when cache utilization reaches said high utilization threshold.
4 . The method of claim 1 further comprising periodically saving write entries in said cache to persistent object storage, and tracking the status of entries in the cache with a first flag to indicate an entry has been written to persistent storage and a second flag to indicate that a corresponding similarity group has been updated.
5 . The method of claim 2 , wherein upon a cache miss occurring when querying for a read entry, the method further comprises modifying the read key to remove the subgroup identifier; re-querying the cache with the modified key; and checking a similarity group that matches the modified key for said subgroup identifier.
6 . The method of claim 5 further comprising, upon there being no matching similarity group, searching object storage using the similarity group identifier as a key; returning a list of matching similarity groups, each similarity group having subgroup identifiers and a transaction identifier that is incremented each time the similarity group is updated; and selecting the similarity group having the highest transaction identifier.
7 . The method of claim 6 further comprising determining whether the selected similarity group has a highest subgroup identifier and, if so, adding the selected similarity group to the write cache with a write key, otherwise adding said selected similarity group to the read cache with a read key.
8 . The method of claim 2 , wherein upon there being a cache miss when querying the cache with said write key, querying object storage using the said write key; searching matching objects for a similarity group having the highest subgroup identifier; and comparing the size of said similarity group having the highest subgroup identifier to a preselected threshold.
9 . The method of claim 8 further comprising, upon said size of said similarity group exceeding said threshold, generating a new similarity group and incrementing a subgroup identifier.
10 . A computer program product comprising non-transitory memory embodying instructions for controlling a processor to perform a method of processing similarity groups for deduplication and restoration in a combined read/write cache comprising an in-memory data structure having a fixed allocation of physical memory that includes a write portion allocated to write entries and a read portion allocated to read entries, the method comprising:
entering similarity group entries into the cache;
accessing said entries in said cache by querying the cache with a key based upon an identifier of a similarity group; and
dynamically varying the sizes of memory allocated to said write portion and to said read portion based upon demand while maintaining said fixed memory allocation constant.
11 . The computer product of claim 10 , wherein said accessing said entries in said cache comprises accessing write entries for deduplication with a write key formed to include an identifier of a similarity group, and accessing read entries for restoration by querying the cache with a read key formed with both an identifier and a subgroup identifier of a similarity group.
12 . The computer product of claim 10 further comprising setting high and low cache utilization thresholds, and evicting least recently used entries from the cache when cache utilization reaches said high utilization threshold.
13 . The computer product of claim 10 further comprising setting high and low cache utilization thresholds, and evicting least recently used entries from the cache when cache utilization reaches said high utilization threshold.
14 . The computer product of claim 10 further comprising periodically writing write entries in said cache to persistent object storage, and tracking the status of entries in the cache with a first flag to indicate an entry has been written to persistent storage and a second flag to indicate that a corresponding similarity group has been updated.
15 . The computer product of claim 11 , wherein upon a cache miss occurring when querying for a read entry, the method further comprises modifying the read key to remove the subgroup identifier; re-querying the cache with the modified key; and checking a similarity group that matches the modified key for said subgroup identifier.
16 . The computer product of claim 15 further comprising, upon there being no matching similarity group, searching object storage using the similarity group identifier as a key; returning a list of matching similarity groups, each similarity group having subgroup identifiers and a transaction identifier that is incremented each time the similarity group is updated; and selecting the similarity group having the highest transaction identifier.
17 . The computer product of claim 16 further comprising determining whether the selected similarity group has a highest subgroup identifier and, if so, adding the selected similarity group to the write cache, otherwise adding said selected similarity group to the read cache.
18 . The computer product of claim 11 , wherein upon there being a cache miss when querying the cache with said write key, querying object storage using the said write key; searching matching objects for a similarity group having the highest subgroup identifier; and comparing the size of said similarity group having the highest subgroup identifier to a preselected threshold.
19 . The computer product of claim 18 further comprising, upon said size of said similarity group exceeding said threshold, generating a new similarity group and incrementing a subgroup identifier.