Parallel deduplication using automatic chunk sizing
An approach for parallel deduplication using automatic chunk sizing. A dynamic chunk deduplicator receives a request to perform data deduplication where the request includes an identification of a dataset. The dynamic chunk deduplicator analyzes file level usage for one or more data files including the dataset to associate a deduplication chunk size with the one or more data files. The dynamic chunk deduplicator creates a collection of data segments from the dataset, based on the deduplication chunk size associated with the one or more data files. The dynamic chunk deduplicator creates a deduplication data chunk size plan where the deduplication data chunk size plan includes deduplication actions for the collection of data segments and outputs the deduplication data chunk size plan.
1. A computer-implemented method for parallel deduplication using automatic chunk sizing, the computer-implemented method comprising:
identifying, by one or more processors, unique and duplicate data segments from one or more data files comprising a data set by comparing hash values of the data segments;
determining, by the one or more processors, a deduplication chunk size corresponding to the duplicate data segments of the one or more data files by analyzing file level usage of the one or more data files comprising the dataset;
creating, by the one or more processors, a collection of data segments from the dataset, based on the deduplication chunk size associated with the one or more data files; and
creating, by the one or more processors, a deduplication data chunk size plan wherein the deduplication data chunk size plan comprises one or more deduplication actions identifying unique data segments of the collection of data segments.
2. The computer-implemented method of claim 1 , further comprising:
determining, by the one or more processors, whether an accept decision of the one or more deduplication actions of the deduplication data chunk size plan is received; and
responsive to receiving the accept decision of the one or more deduplication actions of the deduplication data chunk size plan, creating, by the one or more processors, a collection of parallel deduplication jobs.
3. The computer-implemented method of claim 1 , further comprising:
determining, by the one or more processors, whether a reject decision of the one or more deduplication actions of the deduplication data chunk size plan is received; and
responsive to receiving the reject decision of the one or more deduplication actions of the deduplication data chunk size plan, the one or more processors creating one or more deduplication jobs based on a predetermined chunk size.
4. The computer-implemented method of claim 1 , further comprising:
creating, by the one or more processors, deduplication information wherein the deduplication information comprises at least an index of deduplicated data segments, deduplication chunk size and a deduplication linkage pointer, and wherein the deduplication information is based on at least a collection of parallel deduplication jobs created based on the one or more deduplication actions of the deduplication data chunk size plan; and
storing, by the one or more processors, the deduplication information.
5. The computer-implemented method of claim 1 , wherein the file level usage is based on analyzing an access frequency of the one or more data files, wherein a range of usage thresholds are used to identify respective deduplication chunk size to associate with the one or more data files.
6. The computer-implemented method of claim 2 , wherein one or more jobs associated with the collection of parallel deduplication jobs are grouped into the set of the collection of data segments matching one of the deduplication chunk sizes.
7. The computer-implemented method of claim 1 , wherein a usage threshold is a measure of access frequency of the one or more data files within a predetermined time frame.
8. A computer program product for parallel deduplication using automatic chunk sizing, the computer program product comprising:
one or more non-transitory computer readable storage media and program instructions stored on the one or more non-transitory computer readable storage media, the program instructions comprising:
program instructions to identify unique and duplicate data segments from one or more data files comprising a data set by comparing hash values of the data segments;
program instructions to determine a deduplication chunk size corresponding to the duplicate data segments of the one or more data files by analyzing file level usage of the one or more data files comprising the dataset;
program instructions to create a collection of data segments from the dataset, based on the deduplication chunk size associated with the one or more data files; and
program instructions to create a deduplication data chunk size plan wherein the deduplication data chunk size plan comprises one or more deduplication actions identifying unique data segments of the collection of data segments.
9. The computer program product of claim 8 , further comprising:
program instructions to determine whether an accept decision of the one or more deduplication actions of the deduplication data chunk size plan is received; and
responsive to receiving the accept decision of the one or more deduplication actions of the deduplication data chunk size plan, program instructions to create a collection of parallel deduplication jobs.
10. The computer program product of claim 8 , further comprising:
program instructions to determine whether a reject decision of the one or more deduplication actions of the deduplication data chunk size plan is received; and
responsive to receiving the reject decision of the one or more deduplication actions of the deduplication data chunk size plan, the one or more processors creating one or more deduplication jobs based on a predetermined chunk size.
11. The computer program product of claim 9 , further comprising:
program instructions to create deduplication information wherein the deduplication information comprises at least an index of deduplicated data segments, deduplication chunk size and deduplication linkage pointer, and the deduplication information is based on at least a collection of parallel deduplication jobs created based on the one or more deduplication actions of the deduplication data chunk size plan; and
program instructions to store the deduplication information.
12. The computer-implemented computer program product of claim 8 , wherein the file level usage is based on analyzing access of the one or more data files wherein a range of usage thresholds are used to identify one of the deduplication chunk size to associate with the one or more data files.
13. The computer-implemented computer program product of claim 9 , wherein the one or more jobs associated with the collection of parallel deduplication jobs are grouped into the set of the collection of data segments matching one of the deduplication chunk sizes.
14. The computer-implemented computer program product of claim 8 , wherein a usage threshold is a measure of access frequency of the one or more data files within a predetermined time frame.
15. A computer system for parallel deduplication using automatic chunk sizing, the computer system comprising:
one or more computer processors;
one or more non-transitory computer readable storage media;
program instructions stored on the one or more computer non-transitory readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising:
program instructions to identify unique and duplicate data segments from one or more data files comprising a data set by comparing hash values of the data segments;
program instructions to determine a deduplication chunk size corresponding to the duplicate data segments of the one or more data files by analyzing file level usage of the one or more data files comprising the dataset;
program instructions to create a collection of data segments from the dataset, based on the deduplication chunk size associated with the one or more data files; and
program instructions to create a deduplication data chunk size plan wherein the deduplication data chunk size plan comprises one or more deduplication actions identifying unique data segments of the collection of data segments.
16. The computer system of claim 15 , further comprising:
program instructions to determine whether an accept decision of the one or more deduplication actions of the deduplication data chunk size plan is received; and
responsive to receiving the accept decision of the one or more deduplication actions of the deduplication data chunk size plan, program instructions to create a collection of parallel deduplication jobs.
17. The computer system of claim 15 , further comprising:
program instructions to determine whether a reject decision of the one or more deduplication actions of the deduplication data chunk size plan is received; and
responsive to receiving the reject decision of the one or more deduplication actions of the deduplication data chunk size plan, the one or more processors creating one or more deduplication jobs based on a predetermined chunk size.
18. The computer system of claim 15 , further comprising:
program instructions to create deduplication information wherein the deduplication information comprises at least an index of deduplicated data segments, deduplication chunk size and deduplication linkage pointer and the deduplication information is based on at least a collection of parallel deduplication jobs created based on the one or more deduplication actions of the deduplication data chunk size plan; and
program instructions to store the deduplication information.
19. The computer system of claim 15 , wherein the file level usage is based on analyzing access of the one or more data files wherein a range of usage thresholds are used to identify one of the deduplication chunk size to associate with the one or more data files.
20. The computer system of claim 16 , wherein one or more jobs associated with the collection of parallel deduplication jobs are grouped into the set of the collection of data segments matching one of the deduplication chunk sizes.