System and method for detecting deduplication opportunities
A method, computer program product, and computer system for identifying a plurality of blocks. At least one heuristic associated with at least a portion of the plurality of blocks may be determined. It may be determined whether at least the portion of the plurality of blocks is a candidate for deduplication based upon, at least in part, the at least one heuristic. At least the portion of the plurality of blocks may be deduplicated based upon, at least in part, the at least one heuristic.
1. A computer-implemented method comprising:
identifying a plurality of blocks;
determining at least one heuristic associated with at least a portion of the plurality of blocks;
determining whether at least the portion of the plurality of blocks is a candidate for deduplication based upon, at least in part, the at least one heuristic;
performing additional data comparison on the candidate prior to deduplication to determine if the candidate for deduplication has duplicate data; and
deduplicating at least the portion of the plurality of blocks based upon, at least in part, the at least one heuristic and the additional data comparison.
2. The computer-implemented method of claim 1 wherein the at least one heuristic includes a distance function associated with at least the portion of the plurality of blocks.
3. The computer-implemented method of claim 2 wherein the at least one heuristic includes a digital entropy value associated with at least the portion of the plurality of blocks resulting from the distance function.
4. The computer-implemented method of claim 2 wherein the distance function includes alphabet encoding.
5. The computer-implemented method of claim 3 wherein at least the portion of the plurality of blocks is the candidate for deduplication when the digital entropy value associated with at least the portion of the plurality of blocks resulting from the distance function is less than a threshold.
6. The computer-implemented method of claim 3 wherein at least the portion of the plurality of blocks is not the candidate for deduplication when the digital entropy value associated with at least the portion of the plurality of blocks resulting from the distance function is more than a threshold.
7. A computer program product residing on a non-transitory computer readable storage medium having a plurality of instructions stored thereon which, when executed across one or more processors, causes at least a portion of the one or more processors to perform operations comprising:
identifying a plurality of blocks;
determining at least one heuristic associated with at least a portion of the plurality of blocks;
determining whether at least the portion of the plurality of blocks is a candidate for deduplication based upon, at least in part, the at least one heuristic;
performing additional data comparison on the candidate prior to deduplication to determine if the candidate for deduplication has duplicate data; and
deduplicating at least the portion of the plurality of blocks based upon, at least in part, the at least one heuristic and the additional data comparison.
8. The computer program product of claim 7 wherein the at least one heuristic includes a distance function associated with at least the portion of the plurality of blocks.
9. The computer program product of claim 8 wherein the at least one heuristic includes a digital entropy value associated with at least the portion of the plurality of blocks resulting from the distance function.
10. The computer program product of claim 8 wherein the distance function includes alphabet encoding.
11. The computer program product of claim 9 wherein at least the portion of the plurality of blocks is the candidate for deduplication when the digital entropy value associated with at least the portion of the plurality of blocks resulting from the distance function is less than a threshold.
12. The computer program product of claim 9 wherein at least the portion of the plurality of blocks is not the candidate for deduplication when the digital entropy value associated with at least the portion of the plurality of blocks resulting from the distance function is more than a threshold.
13. A computing system including one or more processors and one or more memories configured to perform operations comprising:
identifying a plurality of blocks;
determining at least one heuristic associated with at least a portion of the plurality of blocks;
determining whether at least the portion of the plurality of blocks is a candidate for deduplication based upon, at least in part, the at least one heuristic;
performing additional data comparison on the candidate prior to deduplication to determine if the candidate for deduplication has duplicate data; and
deduplicating at least the portion of the plurality of blocks based upon, at least in part, the at least one heuristic and the additional data comparison.
14. The computing system of claim 13 wherein the at least one heuristic includes a distance function associated with at least the portion of the plurality of blocks.
15. The computing system of claim 14 wherein the at least one heuristic includes a digital entropy value associated with at least the portion of the plurality of blocks resulting from the distance function.
16. The computing system of claim 15 wherein at least the portion of the plurality of blocks is the candidate for deduplication when the digital entropy value associated with at least the portion of the plurality of blocks resulting from the distance function is less than a threshold, and wherein at least the portion of the plurality of blocks is not the candidate for deduplication when the digital entropy value associated with at least the portion of the plurality of blocks resulting from the distance function is more than the threshold.
17. The computing system of claim 14 wherein the distance function includes alphabet encoding.