Intelligent deduplication data prefetching
View Patent ↗Deduplication dictionaries are used to maintain data chunk identifier and location pairings in a deduplication system. When access to a particular data chunk is requested, a deduplication dictionary is accessed to determine the location of the data chunk and a datastore is accessed to retrieve the data chunk. However, deduplication dictionaries are large and typically maintained on disk, so dictionary access is expensive. Techniques and mechanisms of the present invention allow prefetches or read aheads of datastore (DS) headers. For example, if a dictionary hit results in datastore DS(X), then headers for DS(X+1), DS(X+2), DS(X+read-ahead-window) are prefetched ahead of time. These datastore headers are cached in memory, and indexed by datastore identifier. Before going to the dictionary, a lookup is first performed in the cached headers to reduce deduplication data access request latency.
1. A method, comprising:
receiving a first input/output (I/O) request at a deduplication system node;
performing a first lookup in a cache before accessing a deduplication dictionary;
accessing the deduplication dictionary to determine a first location having data corresponding to the read I/O request;
maintaining in the cache a plurality of headers for a plurality of datastores related to the first location, wherein the cache is configured to maintain the plurality of headers for the plurality of datastores related to the first location upon receiving the read I/O request associated with a single datastore at the first location;
receiving a second I/O request at the deduplication system node;
performing a second lookup in cache before accessing the deduplication dictionary;
responding to the second I/O request without accessing the deduplication dictionary.
2. The method of claim 1 , wherein a number of headers cached corresponds to a read ahead window.
3. The method of claim 1 , wherein the plurality of datastores are determined to be related to the first location if datastore identifiers of the plurality of datastores are monotonically increasing up to a read ahead window from a first datastore identifier associated with the first location.
4. The method of claim 3 , wherein the read ahead window identifies a number of datastore headers to cache.
5. The method of claim 3 , wherein if accessing the deduplication dictionary identifies datastore X, then headers for datastore (X+1) through datastore (X+read ahead window) are prefetched ahead of time.
6. The method of claim 1 , wherein related datastores are created during ingestion of a particular file.
7. The method of claim 1 , wherein the deduplication dictionary maintains mappings between a plurality of fingerprints and a plurality of storage locations.
8. The method of claim 7 , wherein the plurality of fingerprints are a plurality of hash values corresponding to a plurality of deduplicated data chunks.
9. The method of claim 7 , wherein the plurality of fingerprints are a plurality of checksums corresponding to a plurality of deduplicated data chunks.
10. A system, comprising:
a hardware processor;
an interface configured to receive an first input/output (I/O)request and a second (I/O) request at a deduplication system node;
a deduplication dictionary configured to provide a first location having data corresponding to the first I/O request;
a cache configured to maintain a plurality of headers for a plurality of datastores related to a first location after a first lookup is performed, wherein cache is configured to maintain the plurality of headers for the plurality of datastores related to the first location upon receiving the read I/O request associated with a single datastore at the first location, wherein the cache provides response data for the deduplication system node to respond to the second I/O request without accessing the deduplication dictionary.
11. The system of claim 10 , a number of headers cached corresponds to a read ahead window.
12. The system of claim 10 , wherein the plurality of datastores are determined to be related to the first location if datastore identifiers of the plurality of datastores are monotonically increasing up to a read ahead window from a first datastore identifier associated with the first location.
13. The system of claim 12 , wherein the read ahead window identifies a number of datastore headers to cache.
14. The system of claim 12 , wherein if accessing the deduplication dictionary identifies datastore X, then headers for datastore (X+1) through datastore (X+read ahead window) are prefetched ahead of time.
15. The system of claim 10 , wherein related datastores are created during ingestion of a particular file.
16. The system of claim 10 , wherein the deduplication dictionary maintains mappings between a plurality of fingerprints and a plurality of storage locations.
17. The system of claim 16 , wherein the plurality of fingerprints are a plurality of hash values corresponding to a plurality of deduplicated data chunks.
18. The system of claim 16 , wherein the plurality of fingerprints are a plurality of checksums corresponding to a plurality of deduplicated data chunks.
19. A non-transitory computer readable medium, comprising:
computer code for receiving a first input/output (I/O) request at a deduplication system node;
computer code for performing a first lookup in a cache before accessing a deduplication dictionary;
computer code for accessing the deduplication dictionary to determine a first location having data corresponding to the read I/O request;
computer code for maintaining in the cache a plurality of headers for a plurality of datastores related to the first location, wherein the cache is configured to maintain the plurality of headers for the plurality of datastores related to the first location upon receiving the read I/O request associated with a single datastore at the first location;
computer code for receiving a second I/O request at the deduplication system node;
computer code for performing a second lookup in cache before accessing the deduplication dictionary;
computer code for responding to the second I/O request without accessing the deduplication dictionary.
20. The non-transitory computer readable medium of claim 19 , wherein a number of headers cached corresponds to a read ahead window.