Inline deduplication using stream detection
A method and system is used in managing deduplication of data in storage systems. A first digest for a deduplication candidate is received. At least one stream associated with the deduplication candidate is detected. At least one neighboring digest segment of a first loaded digest segment associated with the at least one stream is loaded. Whether the digest is located in the at least one neighboring digest segment is determined. If the digest is not located in the at least one neighboring digest segment, the digest is processed.
1. A method, comprising:
receiving a digest for a deduplication candidate;
detecting at least one stream associated with the deduplication candidate, wherein the at least one stream is detected based on analyzing a target volume of the deduplication candidate, a target logical unit of the deduplication candidate, a target logical unit identifier of the deduplication candidate, sequential writing of the deduplication candidate, writing targeted to a limited region of logical space associated with the deduplication candidate, or a client host and port identifiers of the deduplication candidate;
loading at least one neighboring digest segment of a first loaded digest segment associated with the at least one stream, wherein the at least one neighboring digest segment and the first digest segment are loaded in an index table associated with the at least one stream, and wherein the neighboring digest segment comprises a digest segment that is located sequentially preceding or sequentially following the first digest segment in the index table;
determining whether the digest is located in the at least one neighboring digest segment; and
based on a negative result of the determining, processing the digest, the processing including:
generating a mask;
determining if the digest qualifies as a sample digest based on the mask; and
based on a positive determination that the digest qualifies as a sample digest, searching for the digest in the index table associated with the at least one stream.
2. The method of claim 1 , wherein the processing further comprises, based on a positive determination that the digest qualifies as a sample digest, adding the digest to the index table associated with the at least one stream.
3. The method of claim 1 , wherein the loading at least one neighboring digest segment is in response to identifying the first loaded digest segment as a duplicate.
4. The method of claim 1 , wherein the first loaded digest segment is in a cache.
5. The method of claim 1 , wherein the loading the at least one neighboring digest segment of the first loaded digest segment comprises evicting at least one previously loaded digest segment.
6. The method of claim 5 , wherein the evicted at least one previously loaded digest segment is a digest segment other than the first loaded digest segment.
7. The method of claim 1 , wherein the loading the at least one neighboring digest segment of the first loaded digest segment is performed in response to loading the first loaded digest segment.
8. The method of claim 1 , wherein the processing further comprises, based on a determination that the digest does not qualify as a sample digest, loading the digest into a cache.
9. A system, comprising:
one or more processors; and
computer-executable program logic operating in memory, wherein the computer executable program logic enables execution across the one or more processors of:
receiving a digest for a deduplication candidate;
detecting at least one stream associated with the deduplication candidate, wherein the at least one stream is detected based on analyzing a target volume of the deduplication candidate, a target logical unit of the deduplication candidate, a target logical unit identifier of the deduplication candidate, sequential writing of the deduplication candidate, writing targeted to a limited region of logical space associated with the deduplication candidate, or a client host and port identifiers of the deduplication candidate;
loading at least one neighboring digest segment of a first loaded digest segment associated with the at least one stream, wherein the at least one neighboring digest segment and the first digest segment are loaded in an index table associated with the at least one stream, and wherein the neighboring digest segment comprises a digest segment that is located sequentially preceding or sequentially following the first digest segment in the index table;
determining whether the digest is located in the at least one neighboring digest segment; and
based on a negative result of the determining, processing the digest, the processing including:
generating a mask;
determining if the digest qualifies as a sample digest based on the mask; and
based on a positive determination that the digest qualifies as a sample digest, searching for the digest in the index table associated with the at least one stream.
10. The system of claim 9 , wherein the processing further comprises, based on a positive determination that the digest qualifies as a sample digest, adding the digest to the index table associated with the at least one stream.
11. The system of claim 9 , wherein the loading at least one neighboring digest segment is in response to identifying the first loaded digest segment as a duplicate.
12. The system of claim 9 , wherein the first loaded digest segment is in a cache.
13. The system of claim 9 , wherein the loading the at least one neighboring digest segment of the first loaded digest segment comprises evicting at least one previously loaded digest segment.
14. The system of claim 13 , wherein the evicted at least one previously loaded digest segment is a digest segment other than the first loaded digest segment.
15. The system of claim 9 , wherein the loading the at least one neighboring digest segment of the first loaded digest segment is performed in response to loading the first loaded digest segment.
16. The system of claim 9 , wherein the processing further comprises, based on a determination that the digest does not qualify as a sample digest, loading the digest into a cache.
17. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps:
receiving a digest for a deduplication candidate;
detecting at least one stream associated with the deduplication candidate, wherein the at least one stream is detected based on analyzing a target volume of the deduplication candidate, a target logical unit of the deduplication candidate, a target logical unit identifier of the deduplication candidate, sequential writing of the deduplication candidate, writing targeted to a limited region of logical space associated with the deduplication candidate, or a client host and port identifiers of the deduplication candidate;
loading at least one neighboring digest segment of a first loaded digest segment associated with the at least one stream, wherein the at least one neighboring digest segment and the first digest segment are loaded in an index table associated with the at least one stream, and wherein the neighboring digest segment comprises a digest segment that is located sequentially preceding or sequentially following the first digest segment in the index table;
determining whether the digest is located in the at least one neighboring digest segment; and
based on a negative result of the determining, processing the digest, the processing including:
generating a mask;
determining if the digest qualifies as a sample digest based on the mask; and
based on a positive determination that the digest qualifies as a sample digest, searching for the digest in the index table associated with the at least one stream.
18. The processor-readable storage medium of claim 17 , wherein the processing further comprises, based on a determination that the digest does not qualify as a sample digest, loading the digest into a cache.
19. The processor-readable storage medium of claim 17 , wherein the processing further comprises, based on a positive determination that the digest qualifies as a sample digest, adding the digest to the index table associated with the at least one stream.
20. The processor-readable storage medium of claim 17 , wherein the loading at least one neighboring digest segment is in response to identifying the first loaded digest segment as a duplicate.