Dynamic de-duplication methodologies for efficient resource utilization on de-duplication system
Embodiments are described for reducing memory usage and disk I/O's for reading and writing streams in a deduplication storage system. Generations of backups of a stream can be tracked for the stream. In the first generation (first backup), stream data is ingested, segmented, and written to disk without looking up a fingerprint of each segment, to save disk I/O's and memory. After the first generation, an amount of stream data processed is tracked, and at dynamically tunable increments of data processed, a fingerprint index lookup is performed to disk. The fingerprint index is read into memory and subsequent fingerprint lookups are preformed to cache, until a next dynamically tunable fingerprint lookup to disk. The dynamically tunable increment of processed data can be randomly reset.
1. A computer-implemented method comprising:
determining that a previous generation of stream of data has been deduplicated and written to a storage device as a plurality of segments of data, each segment having a fingerprint;
determining a hit ratio of fingerprint lookups in a fingerprint index cache of the segment storage device, over at least a subset of the plurality of segments of data of the previous generation of the stream of data;
for a subsequent generation of the stream of data being written to the storage device, determining a first fingerprint index lookup threshold that specifies a first amount of data that will be read from the subsequent stream, after which a fingerprint index lookup to the segment storage device will be performed, the index lookup threshold based at least in part on the hit ratio of the segment fingerprint lookups in the fingerprint index cache of the previous generation of the stream.
2. The method of claim 1 , further comprising:
in response to determining that the hit ratio of fingerprint lookups in the fingerprint index cache of the segment storage device for the previous generation of the stream is greater than 8%, setting the fingerprint index lookup threshold to 1 megabyte (1 MB) for deduplicating the subsequent generation of the stream of data.
3. The method claim 1 , further comprising:
receiving data from the subsequent generation of the stream of data;
segmenting the received data into a plurality of segments, and generating a fingerprint for each segment in the plurality of segments;
looking up the fingerprint of the first segment in a fingerprint index on a segment storage device, thereby reading a plurality of fingerprints from the segment storage device into a cached copy of the fingerprint index;
wherein presence of a fingerprint in the fingerprint index indicates that the segment corresponding to the fingerprint has been previously stored on the segment storage device.
4. The method of claim 3 , further comprising:
looking up the fingerprint of the second, and subsequent, segments in the plurality of segments, in the cached copy of the fingerprint index;
wherein presence of a fingerprint in the fingerprint index indicates that the segment corresponding to the fingerprint has been previously stored on the segment storage device.
5. The method of claim 4 , further comprising:
in response to determining that the fingerprint is present in the fingerprint index or in the segment storage cache:
writing a reference to the segment having the fingerprint, to the segment storage device;
increasing a count of fingerprint lookup hits;
otherwise:
writing the segment corresponding to the fingerprint to the segment storage device;
writing the fingerprint, and a reference to the segment, to the fingerprint index; and
inserting the fingerprint, and reference to the segment, into the cached copy of the fingerprint index.
6. The method of claim 1 , further comprising, after deduplicating a threshold amount of received data for the subsequent generation of the stream:
computing a fingerprint cache hit ratio, as a ratio of cache hits to cache lookups;
determining a second fingerprint lookup index threshold that specifies a second amount of data that will be read from the subsequent stream, after which a fingerprint index lookup will be performed, the second fingerprint lookup index based at least in part on the hit ratio of the segment fingerprint lookups in the index cache of the subsequent stream, over the first amount of data.
7. The method of claim 1 , further comprising, in response to determining that a write to storage is not a sequential stream write:
writing the segment to a storage container associated with a stream identifier of the stream of data;
writing the segment fingerprint to a fingerprint index in the storage container associated with the stream identifier of the stream of data.
8. The method of claim 1 , wherein the previous generation of the stream of data was the first generation of the stream of data, and the previous generation of the stream of data was written to segment storage fingerprint without deduplication.
9. A non-transitory computer-readable medium programmed with executable instructions that, when executed by a processing system having at least one hardware processor, perform operations comprising:
determining that a previous generation of stream of data has been deduplicated and written to a storage device as a plurality of segments of data, each segment having a fingerprint;
determining a hit ratio of fingerprint lookups in a fingerprint index cache of the segment storage device, over at least a subset of the plurality of segments of data of the previous generation of the stream of data;
for a subsequent generation of the stream of data being written to the storage device, determining a first fingerprint index lookup threshold that specifies a first amount of data that will be read from the subsequent stream, after which a fingerprint index lookup to the segment storage device will be performed, the index lookup threshold based at least in part on the hit ratio of the segment fingerprint lookups in the fingerprint index cache of the previous generation of the stream.
10. The medium of claim 9 , further comprising:
in response to determining that the hit ratio of fingerprint lookups in the fingerprint index cache of the segment storage device for the previous generation of the stream is greater than 8%, setting the fingerprint index lookup threshold to 1 megabyte (1 MB) for deduplicating the subsequent generation of the stream of data.
11. The medium claim 9 , further comprising:
receiving data from the subsequent generation of the stream of data;
segmenting the received data into a plurality of segments, and generating a fingerprint for each segment in the plurality of segments;
looking up the fingerprint of the first segment in a fingerprint index on a segment storage device, thereby reading a plurality of fingerprints from the segment storage device into a cached copy of the fingerprint index;
wherein presence of a fingerprint in the fingerprint index indicates that the segment corresponding to the fingerprint has been previously stored on the segment storage device.
12. The medium of claim 11 , further comprising:
looking up the fingerprint of the second, and subsequent, segments in the plurality of segments, in the cached copy of the fingerprint index;
wherein presence of a fingerprint in the fingerprint index indicates that the segment corresponding to the fingerprint has been previously stored on the segment storage device.
13. The medium of claim 12 , further comprising:
in response to determining that the fingerprint is present in the fingerprint index or in the segment storage cache:
writing a reference to the segment having the fingerprint, to the segment storage device;
increasing a count of fingerprint lookup hits;
otherwise:
writing the segment corresponding to the fingerprint to the segment storage device;
writing the fingerprint, and a reference to the segment, to the fingerprint index; and
inserting the fingerprint, and reference to the segment, into the cached copy of the fingerprint index.
14. The medium of claim 9 , further comprising, after deduplicating a threshold amount of received data for the subsequent generation of the stream:
computing a fingerprint cache hit ratio, as a ratio of cache hits to cache lookups;
determining a second fingerprint lookup index threshold that specifies a second amount of data that will be read from the subsequent stream, after which a fingerprint index lookup will be performed, the second fingerprint lookup index based at least in part on the hit ratio of the segment fingerprint lookups in the index cache of the subsequent stream, over the first amount of data.
15. The medium of claim 9 , further comprising, in response to determining a write to storage is not a sequential stream write:
writing the segment to a storage container associated with a stream identifier of the stream of data;
writing the segment fingerprint to a fingerprint index in the storage container associated with the stream identifier of the stream of data.
16. The medium of claim 9 , wherein the previous generation of the stream of data was the first generation of the stream of data, and the previous generation of the stream of data was written to segment storage fingerprint without deduplication.
17. A system comprising:
a processing system having at least one hardware processor, the processing system coupled to a memory programmed with executable instructions that, when executed by the processing system, perform operations comprising:
determining that a previous generation of stream of data has been deduplicated and written to a storage device as a plurality of segments of data, each segment having a fingerprint;
determining a hit ratio of fingerprint lookups in a fingerprint index cache of a the segment storage device, over at least a subset of the plurality of segments of data of the previous generation of the stream of data;
for a subsequent generation of the stream of data being written to the storage device, determining a first fingerprint index lookup threshold that specifies a first amount of data that will be read from the subsequent stream, after which a fingerprint index lookup to the segment storage device will be performed, the index lookup threshold based at least in part on the hit ratio of the segment fingerprint lookups in the fingerprint index cache of the previous generation of the stream.
18. The system of claim 17 , further comprising:
in response to determining that the hit ratio of fingerprint lookups in the fingerprint index cache of the segment storage device for the previous generation of the stream is greater than 8%, setting the fingerprint index lookup threshold to 1 megabyte (1 MB) for deduplicating the subsequent generation of the stream of data.
19. The system claim 17 , further comprising:
receiving data from the subsequent generation of the stream of data;
segmenting the received data into a plurality of segments, and generating a fingerprint for each segment in the plurality of segments;
looking up the fingerprint of the first segment in a fingerprint index on a segment storage device, thereby reading a plurality of fingerprints from the segment storage device into a cached copy of the fingerprint index;
wherein presence of a fingerprint in the fingerprint index indicates that the segment corresponding to the fingerprint has been previously stored on the segment storage device.
20. The system of claim 19 , further comprising:
looking up the fingerprint of the second, and subsequent, segments in the plurality of segments, in the cached copy of the fingerprint index;
wherein presence of a fingerprint in the fingerprint index indicates that the segment corresponding to the fingerprint has been previously stored on the segment storage device.
21. The system of claim 20 , further comprising:
in response to determining that the fingerprint is present in the fingerprint index or in the segment storage cache:
writing a reference to the segment having the fingerprint, to the segment storage device;
increasing a count of fingerprint lookup hits;
otherwise:
writing the segment corresponding to the fingerprint to the segment storage device;
writing the fingerprint, and a reference to the segment, to the fingerprint index; and
inserting the fingerprint, and reference to the segment, into the cached copy of the fingerprint index.
22. The system of claim 17 , further comprising, after deduplicating a threshold amount of received data for the subsequent generation of the stream:
computing a fingerprint cache hit ratio, as a ratio of cache hits to cache lookups;
determining a second fingerprint lookup index threshold that specifies a second amount of data that will be read from the subsequent stream, after which a fingerprint index lookup will be performed, the second fingerprint lookup index based at least in part on the hit ratio of the segment fingerprint lookups in the index cache of the subsequent stream, over the first amount of data.
23. The system of claim 17 , further comprising, in response to determining a write to storage is not a sequential stream write:
writing the segment to a storage container associated with a stream identifier of the stream of data;
writing the segment fingerprint to a fingerprint index in the storage container associated with the stream identifier of the stream of data.
24. The system of claim 17 , wherein the previous generation of the stream of data was the first generation of the stream of data, and the previous generation of the stream of data was written to segment storage fingerprint without deduplication.