Accessing deduplicated data from write-evict units in solid-state memory cache
According to some embodiment, a backup storage system receives a request from a client to read a data segment associated with a file object stored in a storage system. In response to the request, the system performs a lookup operation in a first index stored in a memory to identify a first index entry based on a fingerprint of the requested data segment to obtain a first write-evict unit (WEU) identifier (ID) identifying a first WEU storing the requested data segment. The system accesses a solid state device (SSD) operating as a cache memory device to retrieve the data segment from the first WEU. The system extracts and decompresses a compressed data segment retrieved from the first WEU and returns the decompressed data segment to the client without accessing a storage unit for retrieving the same data segment.
1. A computer-implemented method for accessing data segments stored in a storage system, the method comprising:
receiving a request from a client for reading a data segment associated with a file object stored in a storage system;
in response to the request, performing a lookup operation in a first index stored in a memory to identify a first index entry based on a fingerprint of the requested data segment to obtain a first write-evict unit (WEU) identifier (ID) identifying a first WEU storing the requested data segment, wherein the data segment for which the first index entry exists is associated with a frequently accessed file region of the file object, was stored in a storage unit of the storage system prior to a previous client request for writing the data segment, and was written in the first WEU stored in a solid state device (SSD) operating as a cache memory device when the previous client request for writing the data segment was fulfilled, wherein whether the data segment is frequently accessed is determined based on a histogram of the file object recording access statistics of the file object in a predetermined time period in the past indicating that one or more regions in the file object correspond to the frequently accessed file region, and wherein a histogram bucket corresponds to the frequently accessed file region of the file object and is updated each time the frequently accessed file region is accessed over a life of the file object, such that the histogram plots a hierarchy of the frequently accessed file region, the histogram bucket including a timestamp and types of access for the frequently accessed file region of the file object, wherein the data segment is not written to the SSD in response to determining that the data segment is not associated with the frequently accessed file region;
accessing the SSD to retrieve the data segment from the first WEU;
extracting and decompressing a compressed data segment retrieved from the first WEU; and
returning the decompressed data segment to the client without accessing the storage unit for retrieving the same data segment.
2. The method of claim 1 , further comprising:
in response to determining that the first index entry does not exist, performing a lookup operation in a second index based on the fingerprint of the requested data segment to identify a container ID identifying a container storing the requested data segment in the storage unit of the storage system;
retrieving the data segment from the storage unit from the identified container; and
returning the retrieved data segment to the client.
3. The method of claim 2 , further comprising:
compressing the data segment retrieved from the storage unit;
packing the compressed data segment into a second WEU;
writing the second WEU into the SSD device; and
updating the first index to map the fingerprint of the data segment to a second WEU ID identifying the second WEU.
4. The method of claim 2 , wherein the first index comprises a plurality of index entries, each index entry corresponding to one of a plurality of data segments stored in the SSD device.
5. The method of claim 4 , wherein each index entry of the first index maps a portion of a fingerprint of a segment to a physical storage location of a corresponding WEU in the SSD device.
6. The method of claim 2 , wherein the second index comprises a plurality of index entries, each index entry mapping a fingerprint of a corresponding data segment to a container ID identifying a container of the storage unit storing the corresponding data segment.
7. The method of claim 1 , wherein the SSD device stores a plurality of WEUs, wherein each of the WEUs is written to or evicted from the SSD device altogether.
8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations to sort data elements in a data array, the operations comprising:
receiving a request from a client for reading a data segment associated with a file object stored in a storage system;
in response to the request, performing a lookup operation in a first index stored in a memory to identify a first index entry based on a fingerprint of the requested data segment to obtain a first write-evict unit (WEU) identifier (ID) identifying a first WEU storing the requested data segment, wherein the data segment for which the first index entry exists is associated with a frequently accessed file region of the file object, was stored in a storage unit of the storage system prior to a previous client request for writing the data segment, and was written in the first WEU stored in a solid state device (SSD) operating as a cache memory device when the previous client request for writing the data segment was fulfilled, wherein whether the data segment is frequently accessed is determined based on a histogram of the file object recording access statistics of the file object in a predetermined time period in the past indicating that one or more regions in the file object correspond to the frequently accessed file region, and wherein a histogram bucket corresponds to the frequently accessed file region of the file object and is updated each time the frequently accessed file region is accessed over a life of the file object, such that the histogram plots a hierarchy of the frequently accessed file region, the histogram bucket including a timestamp and types of access for the frequently accessed file region of the file object, wherein the data segment is not written to the SSD in response to determining that the data segment is not associated with the frequently accessed file region; accessing the SSD to retrieve the data segment from the first WEU;
extracting and decompressing a compressed data segment retrieved from the first WEU; and
returning the decompressed data segment to the client without accessing the storage unit for retrieving the same data segment.
9. The non-transitory machine-readable medium of claim 8 , further comprising:
in response to determining that the first index entry does not exist, performing a lookup operation in a second index based on the fingerprint of the requested data segment to identify a container ID identifying a container storing the requested data segment in the storage unit of the storage system;
retrieving the data segment from the storage unit from the identified container; and
returning the retrieved data segment to the client.
10. The non-transitory machine-readable medium of claim 9 , further comprising:
compressing the data segment retrieved from the storage unit;
packing the compressed data segment into a second WEU;
writing the second WEU into the SSD device; and
updating the first index to map the fingerprint of the data segment to a second WEU ID identifying the second WEU.
11. The non-transitory machine-readable medium of claim 9 , wherein the first index comprises a plurality of index entries, each index entry corresponding to one of a plurality of data segments stored in the SSD device.
12. The non-transitory machine-readable medium of claim 11 , wherein each index entry of the first index maps a portion of a fingerprint of a segment to a physical storage location of a corresponding WEU in the SSD device.
13. The non-transitory machine-readable medium of claim 9 , wherein the second index comprises a plurality of index entries, each index entry mapping a fingerprint of a corresponding data segment to a container ID identifying a container of the storage unit storing the corresponding data segment.
14. The non-transitory machine-readable medium of claim 8 , wherein the SSD device stores a plurality of WEUs, wherein each of the WEUs is written to or evicted from the SSD device altogether.
15. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
receiving a request from a client for reading a data segment associated with a file object stored in a storage system;
in response to the request, performing a lookup operation in a first index stored in a memory to identify a first index entry based on a fingerprint of the requested data segment to obtain a first write-evict unit (WEU) identifier (ID) identifying a first WEU storing the requested data segment, wherein the data segment for which the first index entry exists is associated with a frequently accessed file region of the file object, was stored in a storage unit of the storage system prior to a previous client request for writing the data segment, and was written in the first WEU stored in a solid state device (SSD) operating as a cache memory device when the previous client request for writing the data segment was fulfilled, wherein whether the data segment is frequently accessed is determined based on a histogram of the file object recording access statistics of the file object in a predetermined time period in the past indicating that one or more regions in the file object correspond to the frequently accessed file region, and wherein a histogram bucket corresponds to the frequently accessed file region of the file object and is updated each time the file region is accessed over a life of the file object, such that the histogram plots a hierarchy of the frequently accessed file region, the histogram bucket including a timestamp and types of access for each of the frequently accessed file region of the file object, wherein the data segment is not written to the SSD in response to determining that the data segment is not associated with the frequently accessed file region;
accessing the SSD to retrieve the data segment from the first WEU;
extracting and decompressing a compressed data segment retrieved from the first WEU; and
returning the decompressed data segment to the client without accessing the storage unit for retrieving the same data segment.
16. The data processing system of claim 15 , further comprising:
in response to determining that the first index entry does not exist, performing a lookup operation in a second index based on the fingerprint of the requested data segment to identify a container ID identifying a container storing the requested data segment in the storage unit of the storage system;
retrieving the data segment from the storage unit from the identified container; and
returning the retrieved data segment to the client.
17. The data processing system of claim 16 , further comprising:
compressing the data segment retrieved from the storage unit;
packing the compressed data segment into a second WEU;
writing the second WEU into the SSD device; and
updating the first index to map the fingerprint of the data segment to a second WEU ID identifying the second WEU.
18. The data processing system of claim 16 , wherein the first index comprises a plurality of index entries, each index entry corresponding to one of a plurality of data segments stored in the SSD device.
19. The data processing system of claim 18 , wherein each index entry of the first index maps a portion of a fingerprint of a segment to a physical storage location of a corresponding WEU in the SSD device.
20. The data processing system of claim 16 , wherein the second index comprises a plurality of index entries, each index entry mapping a fingerprint of a corresponding data segment to a container ID identifying a container of the storage unit storing the corresponding data segment.
21. The data processing system of claim 15 , wherein the SSD device stores a plurality of WEUs, wherein each of the WEUs is written to or evicted from the SSD device altogether.