Method and system for implementing performance tier de-duplication in a virtualization environment
The present application provides an improved approach for managing performance tier de-duplication in a virtualization environment. A content cache is implemented on high performance tiers of storage in order to maintain a working set for the user virtual machines accessing the system, and associates fingerprints with data stored therein. During write requests from the user virtual machines, fingerprints are calculated for the data to be written. However, no de-duplication is performed during the write. During read requests, fingerprints corresponding to the data to be read are retrieved and matched with the fingerprints associated with the data in the content cache. Thus, while multiple pieces of data having the same fingerprints may be written to the lower performance tiers of storage, only one of those pieces of data having that fingerprint will be stored in the content cache for fulfilling read requests.
1. A computer-implemented method for de-duplication, comprising:
receiving a read request from one or more virtual machines to read at least a portion of data;
identifying fingerprints corresponding to the portion of data to be read;
accessing a content cache that stores a set of data for the one or more virtual machines and fingerprints corresponding to the cached set of data, wherein the content cache is configured such that it does not store more than one copy of data corresponding to a particular fingerprint;
determining if any of the identified fingerprints corresponding to the portion of data being requested match any of the fingerprints corresponding to the cached set of data;
if a match is found, retrieving data corresponding to the matched fingerprints from the cached set of data; and
if a match is not found, retrieving the portion of data from an extent store and writing a copy of the portion of data into the content cache, wherein the extent store is a persistent storage device associated with one or more virtual disks.
2. The computer-implemented method of claim 1 , wherein the fingerprints comprise SHA1 fingerprints.
3. The computer-implemented method of claim 1 , wherein the fingerprints are associated with locations on the one or more virtual disks.
4. The computer-implemented method of claim 1 , wherein the content cache is implemented in RAM or flash.
5. The computer-implemented method of claim 1 , wherein the content cache associates fingerprints with working set of data using a hash table.
6. The computer-implemented method of claim 1 , wherein the content cache comprises one or more LRU chains.
7. The computer-implemented method of claim 6 , wherein the LRU chains comprise a single-touch LRU chain and a multi-touch LRU chain.
8. A system for de-duplication in a virtualization environment, comprising:
a processor;
a memory comprising computer code executed using the processor, in which the computer code implements:
receiving a read request from one or more virtual machines to read at least a portion of data;
identifying fingerprints corresponding to the portion of data to be read;
accessing a content cache that stores a set of data for the one or more virtual machines and fingerprints corresponding to the cached set of data, wherein the content cache is configured such that it does not store more than one copy of data corresponding to a particular fingerprint;
determining if any of the identified fingerprints corresponding to the portion of data being requested match any of the fingerprints corresponding to the cached set of data;
if a match is found, retrieving data corresponding to the matched fingerprints from the cached set of data; and
if a match is not found, retrieving the portion of data from an extent store and writing a copy of the portion of data into the content cache, wherein the extent store is a persistent storage device associated with one or more virtual disks.
9. The system of claim 8 , wherein the fingerprints comprise SHA1 fingerprints.
10. The system of claim 8 , wherein the fingerprints are associated with locations on the one or more virtual disks.
11. The system of claim 8 , wherein the content cache is implemented in RAM or flash.
12. The system of claim 8 , wherein the content cache associates fingerprints with working set of data using a hash table.
13. The system of claim 8 , wherein the content cache comprises one or more LRU chains.
14. The system of claim 13 , wherein the LRU chains comprise a single-touch LRU chain and a multi-touch LRU chain.
15. A computer program product embodied on a non-transitory computer readable medium, the computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for de-duplication in a virtualization environment, the method comprising:
receiving a read request from one or more virtual machines to read at least a portion of data;
identifying fingerprints corresponding to the portion of data to be read;
accessing a content cache that stores a set of data for the one or more virtual machines and fingerprints corresponding to the cached set of data, wherein the content cache is configured such that it does not store more than one copy of data corresponding to a particular fingerprint;
determining if any of the identified fingerprints corresponding to the portion of data being requested match any of the fingerprints corresponding to the cached set of data;
if a match is found, retrieving data corresponding to the matched fingerprints from the cached set of data; and
if a match is not found, retrieving the portion of data from an extent store and writing a copy of the portion of data into the content cache, wherein the extent store is a persistent storage device associated with one or more virtual disks.
16. The computer program product of claim 15 , wherein the fingerprints comprise SHA1 fingerprints.
17. The computer program product of claim 15 , wherein the fingerprints are associated with locations on the one or more virtual disks.
18. The computer program product of claim 15 , wherein the content cache is implemented in RAM or flash.
19. The computer program product of claim 15 , wherein the content cache associates fingerprints with working set of data using a hash table.
20. The computer program product of claim 15 , wherein the content cache comprises one or more LRU chains.
21. The computer program product of claim 20 , wherein the LRU chains comprise a single-touch LRU chain and a multi-touch LRU chain.