Poor deduplication identification
Identifying files that do not deduplicate well in a storage system with deduplication facilitates optimizing storage capacity by moving the identified files to less expensive storage without deduplication. Any set of files can be examined to remove files that are identified as files that do not deduplicate well. The process of identification includes arranging the files in a predefined order and using bitmap representations of the unique segments in the files to determine a count of different segments in neighboring next files compared to the previous files, and removing from deduplication any next files that exceed a difference threshold. The bitmap representations of the files allows the identification processes to be performed efficiently for large datasets. Any over-identification of files is minimized by repeating the identification processes on the set of files after arranging them in the reverse order.
1. A computer-implemented method for identifying files not to be deduplicated in a storage system with deduplication, the method comprising:
arranging, by a poor compression module executed by one or more processors, a sequence of files in a predefined order that can be repeated and reversed;
comparing, by the poor compression module, neighboring previous and next files in the arranged sequence of files in the predefined order to determine a count of different segments in the next file compared to the previous files in the sequence;
classifying, by the poor compression module, the next file as a possible file to remove from deduplication if a percentage of the determined count of different segments in the next file exceeds a difference threshold;
reversing, by the poor compression module, the predefined order in which the sequence of files is arranged;
comparing, by the poor compression module, neighboring previous and next files in the arranged sequence of files in the reversed order to again determine the count of different segments in the next file compared to segments in the previous files in the sequence;
declassifying, by the poor compression module, the next file as the possible file to remove from deduplication if the percentage of the again determined count of different segments in the next file is within the difference threshold; and
deduplicating, by a deduplication logic executed by the one or more processors, files which are not classified as possible files to remove from deduplication.
2. The method of claim 1 , wherein arranging the sequence of files in the predefined order that can be repeated and reversed is obtained using a BTree to index inodes of the sequence of files to define a persisted sequence of the files in the pre-defined order.
3. The computer-implemented method of claim 1 , wherein segments in the next file and previous file(s) are counted based on bitmaps generated for each file in the sequence of files, the bitmaps representing segments belonging to the files in memory in a countable set of bits having a binary value of one or zero.
4. The computer-implemented method of claim 3 , wherein to determine the count of different segments in the next file compared to segments in the previous file includes:
generating a combined bitmap representing unique segments collectively belonging to the neighboring previous and next files in the sequence of files; and
subtracting a count of a number of unique segments represented in the previous file's bitmap from a count of a number of unique segments represented in the combined bitmap to reveal the count of different segments in the next file compared to the previous file.
5. The computer-implemented method of claim 4 , wherein generating the combined bitmap representing unique segments collectively belonging to the neighboring previous and next files in the sequence of files is taking a union of a bitmap for the previous file and the next file.
6. The computer-implemented method of claim 3 further comprising generating the bitmap using a bloom filter, the generating including:
receiving a selection of the N files for which files that do not duplicate are to be identified;
determining that a file is one of the files in the N files;
traversing segments belonging to the file;
sampling one of every R segments traversed; and
inserting each sampled one of every R segments into the bitmap using the bloom filter.
7. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for identifying files that do not deduplicate in a storage system with deduplication, the operations comprising:
arrange a sequence of files in a predefined order that can be repeated and reversed;
compare neighboring previous and next files in the arranged sequence of files in the predefined order to determine a count of different segments in the next file compared to the previous files in the sequence;
classify the next file as a possible file to remove from deduplication if a percentage of the determined count of different segments in the next file exceeds a difference threshold;
reverse the predefined order in which the sequence of files is arranged;
compare neighboring previous and next files in the arranged sequence of files in the reversed order to again determine the count of different segments in the next file compared to segments in the previous files in the sequence;
declassify the next file as the possible file to remove from deduplication if the percentage of the again determined count of different segments in the next file is within the difference threshold; and
deduplicate files which are not classified as possible files to remove from deduplication.
8. The non-transitory computer-readable storage medium of claim 7 , wherein to arrange the sequence of files in the predefined order that can be repeated and reversed is to obtain a BTree to index inodes of the sequence of files to define a persisted sequence of the files in the pre-defined order.
9. The non-transitory computer-readable storage medium of claim 7 , wherein segments in the next file and previous file(s) are counted based on bitmaps generated for each file in the sequence of files, the bitmaps representing segments belonging to the files in memory in a countable set of bits having a binary value of one or zero.
10. The non-transitory computer-readable storage medium of claim 7 , wherein to determine the count of different segments in the next file compared to segments in the previous file(s) includes to:
generate a combined bitmap representing unique segments collectively belonging to the neighboring previous and next files in the sequence of files; and
subtract a count of a number of unique segments represented in the previous file's bitmap from a count of a number of unique segments represented in the combined bitmap to reveal the count of different segments in the next file compared to the previous file(s).
11. The non-transitory computer-readable storage medium of claim 10 , wherein to generate the combined bitmap representing unique segments collectively belonging to the neighboring previous and next files in the sequence of files is taking a union of a bitmap for the previous file(s) and the next file.
12. The non-transitory computer-readable storage medium of claim 9 , further comprising operations to:
generate the bitmap using a bloom filter, including to:
receive a selection of the N files for which files that do not duplicate are to be identified;
determine that a file is one of the files in the N files;
traverse segments belonging to the file;
sample one of every R segments traversed; and
insert each sampled one of every R segments into the bitmap using the bloom filter.
13. A data processing system, comprising:
a memory in which to store a bitmap for each file in a subset of files, and a repository for storing information identifying which files in the subset of files that do not compress;
a processor in communication with the memory, the processor configured to:
arrange a sequence of files in a predefined order that can be repeated and reversed;
compare neighboring previous and next files in the arranged sequence of files in the predefined order to determine a count of different segments in the next file compared to the previous files in the sequence;
classify the next file as a possible file to remove from deduplication if a percentage of the determined count of different segments in the next file exceeds a difference threshold;
reverse the predefined order in which the sequence of files is arranged;
compare neighboring previous and next files in the arranged sequence of files in the reversed order to again determine the count of different segments in the next file compared to segments in the previous files in the sequence;
declassify the next file as the possible file to remove from deduplication if the percentage of the again determined count of different segments in the next file is within the difference threshold; and
deduplicate files which are not classified as possible files to remove from deduplication.
14. The data processing system of claim 13 , wherein to arrange the sequence of files in the predefined order that can be repeated and reversed is to obtain a BTree to index inodes of the sequence of files to define a persisted sequence of the files in the pre-defined order.
15. The data processing system of claim 13 , wherein to count segments in the next file and previous file(s) is based on bitmaps generated for each file in the sequence of files, the bitmaps representing segments belonging to the files in memory in a countable set of bits having a binary value of one or zero.
16. The data processing system of claim 15 , wherein to determine the count of different segments in the next file as compared to segments in the previous file, the processor is to further:
generate a combined bitmap representing unique segments collectively belonging to the neighboring previous and next files in the sequence of files; and
subtract a count of a number of unique segments represented in the previous file's bitmap from a count of a number of unique segments represented in the combined bitmap to reveal the count of different segments in the next file compared to the previous file.
17. The data processing system of claim 16 , wherein to generate the combined bitmap representing unique segments collectively belonging to the neighboring previous and next files in the sequence of files, the processor is to take a union of a bitmap for the previous file and the next file.
18. The data processing system of claim 15 , wherein the processor is further to generate the bitmap using a bloom filter, including to:
receive a selection of the N files for which files that do not duplicate are to be identified;
determine that a file is one of the files in the N files;
traverse segments belonging to the file;
sample one of every R segments traversed; and
insert each sampled one of every R segments into the bitmap using the bloom filter.