System and method for granular deduplication
A method of managing a memory system is described, the memory system having a processor with a client interface, a buffer memory, and a storage memory. A LUN (logical unit) is exposed to the client, the LUN having a memory size, and QoS specification. When the processor receives a command from the client to write data to the LUN, determining if the QoS specification includes deduplication and: if the QoS does not include deduplication, processing the command by storing the data in the storage memory and creating metadata uniquely referencing a storage location of the data that is written; or if the QoS includes deduplication, determine if deduplication is possible while meeting a latency value as specified in the QoS specification and performing deduplication.
1. A method comprising:
receiving a command to write data to a logical unit (LUN) of memory, wherein the LUN is associated with a quality of service (QoS) specification;
determining that the QoS specification associated with the LUN includes deduplication;
determining whether deduplication is possible while meeting the QoS specification;
deduplicating and storing the data in a storage memory in response to a determination that deduplication is possible while meeting the QoS specification; and
storing the data in the storage memory and creating metadata referencing a storage location of the data in response to a determination that deduplication is not possible while meeting the QoS specification.
2. The method of claim 1 , further comprising:
determining whether compression is supported by the QoS specification associated with the LUN; and
compressing the deduplicated data when storing the data in the storage memory in response to the QoS specification supporting compression.
3. The method of claim 2 , wherein the deduplicated data is compressed when storing the data based on a latency in the QoS specification associated with the LUN.
4. The method of claim 1 , wherein the deduplicating and storing the data in the storage memory is based on a latency included in the QoS specification associated with the LUN.
5. The method of claim 1 , wherein the deduplicating and storing the data in the storage memory is based on free storage capacity available to the LUN.
6. The method of claim 1 , wherein the LUN is allocated to a client that sends the command to write data to the LUN, the LUN being associated with the QoS specification and the LUN having a memory size.
7. The method of claim 1 , wherein the deduplicating and storing the data in the storage memory comprises:
comparing a computed hash value of the data with hash values in a hash value repository of a single instance repository (SIR);
adding a reference to a first storage location in the SIR associated with the data in response to the computed hash value of the data being found in the hash value repository; and
adding the computed hash value to the hash value repository and storing the data in a second storage location in the SIR in response to the computed hash value of the data not being found in the hash value repository.
8. The method of claim 7 , further comprising maintaining the SIR, wherein maintaining the SIR comprises:
evicting a hash value and associated stored data from the SIR in response to the hash value not being referenced by any LUN; and
maintaining a hash value and associated stored data in the SIR in response to the has value being referenced by one or more LUNs.
9. The method of claim 8 , wherein the hash value and associated stored data are evicted from the SIR in response to the hash value not being referenced by any LUN for a predetermined period of time.
10. The method of claim 1 , wherein the LUN of memory is associated with any one of a plurality of attributes including QoS measures, average and maximum read or write latency, bandwidth requirements, and storage capacity.
11. An apparatus comprising:
a storage memory; and
a processor configured to:
receive a command to write data to a logical unit (LUN) of memory, wherein the LUN is associated with a quality of service (QoS) specification;
determine that the QoS specification associated with the LUN includes deduplication;
deduplicate and store the data in a storage memory in response to a determination that deduplication is possible while meeting the QoS specification; and
storing the data in the storage memory and creating metadata referencing a storage location of the data in response to a determination that deduplication is not possible while meeting the QoS specification.
12. The apparatus of claim 11 , wherein the processor is further configured to:
determine whether compression is supported by the QoS specification associated with the LUN; and
compress the deduplicated data when storing the data in the storage memory in response to the QoS specification supporting compression.
13. The apparatus of claim 12 , wherein the deduplicated data is compressed when storing the data based on a latency in the QoS specification associated with the LUN.
14. The apparatus of claim 11 , wherein the processor is further configured to deduplicate and store the data in the storage memory based on a latency included in the QoS specification associated with the LUN.
15. The apparatus of claim 11 , wherein the processor is further configured to deduplicate and store the data in the storage memory based on free storage capacity available to the LUN.
16. The apparatus of claim 11 , wherein the LUN is allocated to a client that sends the command to write data to the LUN, the LUN being associated with the QoS specification and the LUN having a memory size.
17. The apparatus of claim 11 , wherein the processor is further configured to deduplicate and store the data in the storage memory by:
comparing a computed hash value of the data with hash values in a hash value repository of a single instance repository (SIR);
adding a reference to a first storage location in the SIR associated with the data in response to the computed hash value of the data being found in the hash value repository; and
adding the computed hash value to the hash value repository and storing the data in a second storage location in the SIR in response to the computed hash value of the data not being found in the hash value repository.
18. The apparatus of claim 17 , wherein the processor is further configured to maintain the SIR by:
evicting a hash value and associated stored data from the SIR in response to the hash value not being referenced by any LUN; and
maintaining a hash value and associated stored data in the SIR in response to the has value being referenced by one or more LUNs.
19. The apparatus of claim 18 , wherein the hash value and associated stored data are evicted from the SIR in response to the hash value not being referenced by any LUN for a predetermined period of time.
20. The apparatus of claim 11 , wherein the LUN of memory is associated with any one of a plurality of attributes including QoS measures, average and maximum read or write latency, bandwidth requirements, and storage capacity.