System and method for granular deduplication

1. A method Of managing a memory system, comprising:

providing a memory system with a processor having a client interface, a buffer memory, and a storage memory;

allocating a logical unit (LUN) to the client, the LUN having a memory size, and QoS specification;

receiving a command from the client to write data to the LUN; and

determining if the QoS specification includes deduplication and:

if the QoS does not include deduplication, process the command storing the data in the storage memory and creating metadata uniquely referencing a storage location of the data that is written; or

if the QoS includes deduplication, determine if deduplication is possible while meeting a latency value as specified in the QoS specification and:

if deduplication is possible, performing deduplication and creating metadata relating a user address to a physical location where a copy of the data that is written is stored; or,

if deduplication is not possible, storing undeduplicated data in a backing LUN associated with a deduplicated LUN, and

draining undeduplicated data stored in the backing LUN when deduplication is possible by performing deduplication.

2. A method of managing a memory system, comprising:

providing a memory system with a processor having a client interface, a buffer memory, and a storage memory;

maintaining a metadata relating a hash value to a location in the memory system where the data is stored;

associating a plurality of logical units (LUN), with a SIR (single instance repository), each LUN being exposed to a user and having a stub LUN (SL) relating a user logical block address (LBA) to a hash value of data or to a location in the memory system where the data is stored,

wherein hash values comprise hash values that are referenced by at least one of a plurality of LBAs in a LUN or by a plurality of LUNs; and

the hash values are garbage collected by:

determining whether the hash value is referenced by one or more LUNs and:

in accordance with a policy determined by user quality of service requirements (QoS),

evicting the hash value and associated stored data if the hash value is not referenced by any LUN; or

evicting the hash value and associated stored data if the hash value is not referenced by any LUN for more than one garbage collection cycle; or

evicting the hash value if the hash value is referenced by a LUN and the hash value has not been referenced for a predetermined period of time;

maintaining the hash value and associated stored data if the hash value is referenced by more than one LUN.

3. The method of claim 2 wherein, the metadata is stored in the SIR.