Parallel deduplication using automatic chunk sizing
An approach for parallel deduplication using automatic chunk sizing. A dynamic chunk deduplicator receives a request to perform data deduplication where the request includes an identification of a dataset. The dynamic chunk deduplicator analyzes file level usage for one or more data files including the dataset to associate a deduplication chunk size with the one or more data files. The dynamic chunk deduplicator creates a collection of data segments from the dataset, based on the deduplication chunk size associated with the one or more data files. The dynamic chunk deduplicator creates a deduplication data chunk size plan where the deduplication data chunk size plan includes deduplication actions for the collection of data segments and outputs the deduplication data chunk size plan.
1. A computer-implemented method for parallel deduplication using dynamic chunk sizing, the computer-implemented method comprising:
receiving, by a dynamic chunk deduplicator, a request to perform data deduplication on a dataset that is identified;
computing, by the dynamic chunk deduplicator, a deduplication chunk size for one or more data files of the dataset, the deduplication chunk size based on an analysis of a file level usage of the one or more data files of the dataset, wherein the file level usage is based on analyzing access frequency of the one or more data files, respectively, and wherein a range of usage thresholds are used to identify a deduplication chunk size to associate with the one or more data files, respectively, such that a smaller deduplication chunk size corresponds to one or more data files of the data set having a lower access frequency, and a larger deduplication chunk size corresponds to one or more data files of the dataset having a higher access frequency;
determining, by the dynamic chunk deduplicator, a collection of data segments from the one or more data files of the dataset, wherein a group of data segments of the collection of data segments having a similar deduplication chunk size, wherein hash values of the group of data segments of the collection of data segments are compared to determine duplicate data segments;
creating, by the dynamic chunk deduplicator, deduplication information, wherein the deduplication information comprises at least an index of duplicated data segments, deduplication chunk size, and deduplication linkage pointers, and wherein the deduplication information is based on at least the one or more data files of the dataset;
creating, by the dynamic chunk deduplicator, a deduplication data chunk size plan wherein the deduplication data chunk size plan comprises one or more deduplication actions for the collection of data segments, and wherein the deduplication chunk size plan is transmitted for a decision whether to proceed with the one or more deduplication actions of the deduplication chunk size plan; and
responsive to receiving the decision to perform the one or more deduplication actions of the deduplication chunk size plan, creating, by the dynamic chunk deduplicator, a collection of parallel deduplication jobs comprised of data segments from the one or more data files of the dataset, having similar deduplication chunk size.