Caching of metadata for deduplicated luns
Efficient processing of user data read requests in a deduplicated data storage system places the metadata for most frequently requested data in data structures and locations in the system hierarchy where the metadata will be most rapidly available. The total amount of such metadata makes storing all of the metadata in high speed memory expensive, and the system and method described uses both the temporal and the spatial characteristics of the user system activity in any epoch to adjust the contents of metadata cache so as to respond to the dynamics of a multi user or multi-application environment where the storage system is not made aware of the time changing mix of operations except by observation of the individual requests. A history record is used to promote metadata from the slow memory to the fast memory, and a process selection may be adjusted based on the address-space activity.
1. A computer system for storing data comprising:
a processor configured to:
store metadata for deduplicated data in a non-volatile storage memory;
store a subset of the metadata for the deduplicated data in at least one deduplicated metadata cache in a higher speed memory than the non-volatile storage memory;
search the deduplicated metadata cache;
read the deduplicated data from the non-volatile storage memory using the subset of the metadata from the deduplicated metadata cache when the metadata is stored in the deduplicated metadata cache; and
read the deduplicated data from the storage memory using the metadata from the non-volatile storage memory when the metadata is not stored in the deduplicated metadata cache.
2. The system of claim 1 , wherein the processor is further configurable to:
maintain a history record of accesses to the deduplicated data; and
store the metadata for the deduplicated data in the deduplicated metadata cache based on the history record.
3. The system of claim 2 , wherein the processor is further configurable to:
search the history record; and
increment an access counter value when the metadata is already in the history record.
4. The system of claim 2 , wherein the processor is further configurable to:
search the history record prior to searching of the deduplicated metadata cache; and
search the deduplicated metadata cache for the metadata based on search results of the history record.
5. The system of claim 1 , wherein the processor is further configurable to:
use multiple metadata from the deduplicated metadata cache to read the deduplicated data from multiple locations in the non-volatile storage memory.
6. The system of claim 1 , wherein the processor is further configurable to:
use the metadata from the deduplicated metadata cache to read the deduplicated data from multiple locations in the non-volatile storage memory.
7. The system of claim 1 , wherein the processor is further configurable to:
determine a number of accesses to the metadata in the deduplicated metadata cache;
retain the metadata in the deduplicated metadata cache when the number of accesses exceeds a threshold; and
evict at least some of the other metadata from the deduplicated metadata cache not selected for retention.
8. The system of claim 1 , wherein the processor is further configurable to:
determine an age status of the metadata in the deduplicated metadata cache;
retain the metadata in the deduplicated metadata cache based on the age status; and
evict at least some of the other metadata from the deduplicated metadata cache not selected for retention.
9. The system of claim 1 , wherein the processor is configurable to:
use one or more Bloom filters to identify the metadata in the deduplicated metadata cache.
10. A non-transitory computer-readable storage medium (NTCRSM) comprising a set of instructions, that when executed by a hardware processor, cause the hardware processor to:
store metadata for deduplicated data in a storage memory;
store a subset of the metadata in at least one deduplicated metadata cache;
search the deduplicated metadata cache; and
use the metadata stored in the deduplicated metadata cache to access the deduplicated data from the storage memory.
11. The NTCRSM of claim 10 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
use the metadata from the storage memory to access the deduplicated data from the storage memory when the metadata is not stored in the deduplicated metadata cache.
12. The NTCRSM of claim 11 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
increment an access counter value for the metadata; and
add the metadata to the deduplicated metadata cache, or maintain the metadata in the deduplicated metadata cache, based on the access counter value for the metadata.
13. The NTCRSM of claim 10 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
check the access counter value for the metadata over a time period; and
add or maintain the metadata in the deduplicated data cache when the access counter value for the time period exceeds a threshold.
14. The NTCRSM of claim 13 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
add the metadata to the deduplicated metadata cache and selectively evict other metadata from the deduplicated metadata cache based on available space in the deduplicated metadata cache.
15. The NTCRSM of claim 10 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
determine an age status of the metadata in the deduplicated metadata cache; and
maintain or purge the metadata in the deduplicated metadata cache based on the age status.
16. The NTCRSM of claim 10 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
use a Bloom filter to determine if the metadata is in the deduplicated metadata cache.
17. The NTCRSM of claim 10 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
maintain a history record for the deduplicated data; and
add metadata to the deduplicated metadata cache based on the history record for the deduplicated data.
18. The NTCRSM of claim 17 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
search the history record prior to searching the deduplicated metadata cache.
19. The NTCRSM of claim 17 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
store an address record for the deduplicated data in the history record; and
increment an access counter value when the address record already exists in the history record.
20. The NTCRSM of claim 19 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
mark the address record when the metadata for the deduplicated data is stored in the deduplicated metadata cache.
21. The NTCRSM of claim 19 , wherein the set of instructions, when executed by the hardware processor, further cause the hardware processor to:
search the deduplicated metadata cache or the storage memory for the metadata based on the marked address record.