Intelligent deduplication data prefetching
Deduplication dictionaries are used to maintain data chunk identifier and location pairings in a deduplication system. When access to a particular data chunk is requested, a deduplication dictionary is accessed to determine the location of the data chunk and a datastore is accessed to retrieve the data chunk. However, deduplication dictionaries are large and typically maintained on disk, so dictionary access is expensive. Techniques and mechanisms of the present invention allow prefetches or read aheads of datastore (DS) headers. For example, if a dictionary hit results in datastore DS(X), then headers for DS (X+1), DS (X+2), DS(X+read-ahead-window) are prefetched ahead of time. These datastore headers are cached in memory, and indexed by datastore identifier. Before going to the dictionary, a lookup is first performed in the cached headers to reduce deduplication data access request latency.
1. A method, comprising:
receiving a read input/output (I/O) request at a deduplication system node;
accessing a deduplication dictionary to determine a first location having data corresponding to the read I/O request;
caching headers for a plurality of datastores related to the first location, wherein cache is configured to cache the headers for the plurality of datastores related to the first location upon receiving the read I/O request associated with a single datastore at the first location;
receiving a subsequent read I/O request at the deduplication system node;
performing a lookup in cached headers before accessing any deduplication dictionary;
responding to the subsequent read I/O request.
2. The method of claim 1 , wherein a number of headers cached corresponds to a read ahead window.
3. The method of claim 1 , wherein the plurality of datastores are determined to be related to the first location if datastore identifiers of the plurality of datastores are monotonically increasing up to a read ahead window from a first datastore identifier associated with the first location.
4. The method of claim 3 , wherein the read ahead window identifies a number of datastore headers to cache.
5. The method of claim 3 , wherein if accessing the deduplication dictionary identifies datastore X, then headers for datastore (X+1) through datastore (X+read ahead window) are prefetched ahead of time.
6. The method of claim 1 , wherein related datastores are created during ingestion of a particular file.
7. The method of claim 1 , wherein the deduplication dictionary maintains mappings between a plurality of fingerprints and a plurality of storage locations.
8. The method of claim 7 , wherein the plurality of fingerprints are a plurality of hash values corresponding to a plurality of deduplicated data chunks.
9. The method of claim 7 , wherein the plurality of fingerprints are a plurality of checksums corresponding to a plurality of deduplicated data chunks.
10. A system, comprising:
an interface configured to receive a read input/output (I/O) request at a deduplication system node;
a processor configured to access a deduplication dictionary to determine a first location having data corresponding to the read I/O request;
cache configured to maintain headers for a plurality of datastores related to the first location, wherein cache is configured to cache the headers for the plurality of datastores related to the first location upon receiving the read I/O request associated with a single datastore at the first location;
wherein the interface is operable to receive a subsequent read I/O request at the deduplication system node and respond to the subsequent read I/O request upon performing a lookup in cached headers before accessing any deduplication dictionary.
11. The system of claim 10 , wherein a number of headers cached corresponds to a read ahead window.
12. The system of claim 10 , wherein the plurality of datastores are determined to be related to the first location if datastore identifiers of the plurality of datastores are monotonically increasing up to a read ahead window from a first datastore identifier associated with the first location.
13. The system of claim 12 , wherein the read ahead window identifies a number of datastore headers to cache.
14. The system of claim 12 , wherein if accessing the deduplication dictionary identifies datastore X, then headers for datastore (X+1) through datastore (X+read ahead window) are prefetched ahead of time.
15. The system of claim 10 , wherein related datastores are created during ingestion of a particular file.
16. The system of claim 10 , wherein the deduplication dictionary maintains mappings between a plurality of fingerprints and a plurality of storage locations.
17. The system of claim 16 , wherein the plurality of fingerprints are a plurality of hash values corresponding to a plurality of deduplicated data chunks.
18. The system of claim 16 , wherein the plurality of fingerprints are a plurality of checksums corresponding to a plurality of deduplicated data chunks.
19. A system, comprising:
means for receiving a read input/output (I/O) request at a deduplication system node;
means for accessing a deduplication dictionary to determine a first location having data corresponding to the read I/O request, wherein cache is configured to cache the headers for the plurality of datastores related to the first location upon receiving the read I/O request associated with a single datastore at the first location;
means for caching headers for a plurality of datastores related to the first location;
means for receiving a subsequent read I/O request at the deduplication system node;
means for performing a lookup in cached headers before accessing any deduplication dictionary;
means for responding to the subsequent read I/O request.
20. The system of claim 19 , wherein a number of headers cached corresponds to a read ahead window.