Parallel deduplication using automatic chunk sizing
An approach for parallel deduplication using automatic chunk sizing. A dynamic chunk deduplicator receives a request to perform data deduplication where the request includes an identification of a dataset. The dynamic chunk deduplicator analyzes file level usage for one or more data files including the dataset to associate a deduplication chunk size with the one or more data files. The dynamic chunk deduplicator creates a collection of data segments from the dataset, based on the deduplication chunk size associated with the one or more data files. The dynamic chunk deduplicator creates a deduplication data chunk size plan where the deduplication data chunk size plan includes deduplication actions for the collection of data segments and outputs the deduplication data chunk size plan.
1. A computer-implemented method for parallel deduplication using automatic chunk sizing, the computer-implemented method comprising:
receiving, by a dynamic chunk deduplicator, a request to perform data deduplication wherein the request comprises an identification of a dataset;
analyzing, by the dynamic chunk deduplicator, file level usage for one or more data files comprising the dataset to associate a deduplication chunk size with the one or more data files;
creating, by the dynamic chunk deduplicator, a collection of data segments from the dataset, based on the deduplication chunk size associated with the one or more data files;
creating, by the dynamic chunk deduplicator, a deduplication data chunk size plan wherein the deduplication data chunk size plan comprises one or more deduplication actions for the collection of data segments; and
outputting, by the dynamic chunk deduplicator, the deduplication data chunk size plan.
2. The computer-implemented method of claim 1 , further comprising:
receiving, by the dynamic chunk deduplicator, a plan decision of at least one of an accept or reject of the one or more deduplication actions;
responsive to receiving the plan decision is accept, creating, by the dynamic chunk deduplicator, a collection of parallel deduplication jobs; and
responsive to receiving the plan decision is reject, creating, by the dynamic chunk deduplicator, a deduplication job wherein the one or more deduplication jobs is based on a predetermined chunk size.
3. The computer-implemented method of claim 2 , further comprising:
creating, by the dynamic chunk deduplicator, deduplication information wherein the deduplication information comprises at least an index of deduplicated data segments, deduplication chunk size and deduplication linkage pointer and the deduplication information is based on at least the collection of parallel deduplication jobs or the one or more deduplication jobs; and
storing, by the dynamic chunk deduplicator, the deduplication information.
4. The computer-implemented method of claim 1 , wherein the file level usage is based on analyzing access of the one or more data files wherein a range of usage thresholds are used to identify one of the deduplication chunk size to associate with the one or more data files.
5. The computer-implemented method of claim 1 , wherein the one or more deduplication actions is determined by identifying unique data segments based on hash value comparison of the collection of data segments.
6. The computer-implemented method of claim 2 , wherein one or more jobs associated with the collection of parallel deduplication jobs is based on one of the deduplication chunk size to group a respective set of the collection of data segments matching the deduplication chunk size.
7. The computer-implemented method of claim 4 , wherein a usage threshold is a measure of access frequency of the one or more data files within a predetermined time frame.
8. A computer-implemented computer program product for parallel deduplication using automatic chunk sizing, the computer program product comprising:
one or more non-transitory computer readable storage media and program instructions stored on the one or more non-transitory computer readable storage media, the program instructions comprising:
program instructions to, receive, by a dynamic chunk deduplicator, a request to perform data deduplication wherein the request comprises an identification of a dataset;
program instructions to, analyze, by the dynamic chunk deduplicator, file level usage for one or more data files comprising the dataset to associate a deduplication chunk size with the one or more data files;
program instructions to, create, by the dynamic chunk deduplicator, a collection of data segments from the dataset, based on the deduplication chunk size associated with the one or more data files;
program instructions to, create, by the dynamic chunk deduplicator, a deduplication data chunk size plan wherein the deduplication data chunk size plan comprises one or more deduplication actions for the collection of data segments; and
program instructions to, output, by the dynamic chunk deduplicator, the deduplication data chunk size plan.
9. The computer-implemented computer program product of claim 8 , further comprising:
program instructions to, receive, by the dynamic chunk deduplicator, a plan decision of at least one of an accept or reject of the one or more deduplication actions;
program instructions to, respond to receive the plan decision is accept, creating, by the dynamic chunk deduplicator, a collection of parallel deduplication jobs; and
program instructions to, respond to receive the plan decision is reject, creating, by the dynamic chunk deduplicator, one or more deduplication jobs wherein the one or more deduplication jobs is based on a predetermined chunk size.
10. The computer-implemented computer program product of claim 9 , further comprising:
program instructions to, create, by the dynamic chunk deduplicator, deduplication information wherein the deduplication information comprises at least an index of deduplicated data segments, deduplication chunk size and deduplication linkage pointer and the deduplication information is based on at least the collection of parallel deduplication jobs or the one or more deduplication jobs; and
program instructions to, store, by the dynamic chunk deduplicator, the deduplication information.
11. The computer-implemented computer program product of claim 8 , wherein the file level usage is based on analyzing access of the one or more data files wherein a range of usage thresholds are used to identify one of the deduplication chunk size to associate with the one or more data files.
12. The computer-implemented computer program product of claim 8 , wherein the one or more deduplication actions is determined by identifying unique data segments based on hash value comparison of the collection of data segments.
13. The computer-implemented computer program product of claim 9 , wherein one or more jobs associated with the collection of parallel deduplication jobs is based on one of the deduplication chunk size to group a respective set of the collection of data segments matching the deduplication chunk size.
14. The computer-implemented computer program product of claim 11 , wherein a usage threshold is a measure of access frequency of the one or more data files within a predetermined time frame.
15. A computer system for parallel deduplication using automatic chunk sizing, the computer system comprising:
one or more computer processors;
one or more non-transitory computer readable storage media;
program instructions stored on the one or more computer non-transitory readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising:
program instructions to, receive, by a dynamic chunk deduplicator, a request to perform data deduplication wherein the request comprises an identification of a dataset;
program instructions to, analyze, by the dynamic chunk deduplicator, file level usage for one or more data files comprising the dataset to associate a deduplication chunk size with the one or more data files;
program instructions to, create, by the dynamic chunk deduplicator, a collection of data segments from the dataset, based on the deduplication chunk size associated with the one or more data files;
program instructions to, create, by the dynamic chunk deduplicator, a deduplication data chunk size plan wherein the deduplication data chunk size plan comprises one or more deduplication actions for the collection of data segments; and
program instructions to, output, by the dynamic chunk deduplicator, the deduplication data chunk size plan.
16. The computer system of claim 15 , further comprising:
program instructions to, receive, by the dynamic chunk deduplicator, a plan decision of at least one of an accept or reject of the one or more deduplication actions;
program instructions to, respond to receive the plan decision is accept, creating, by the dynamic chunk deduplicator, a collection of parallel deduplication jobs; and
program instructions to, respond to receive the plan decision is reject, creating, by the dynamic chunk deduplicator, one or more deduplication jobs wherein the one or more deduplication jobs is based on a predetermined chunk size.
17. The computer system of claim 16 , further comprising:
program instructions to, create, by the dynamic chunk deduplicator, deduplication information wherein the deduplication information comprises at least an index of deduplicated data segments, deduplication chunk size and deduplication linkage pointer and the deduplication information is based on at least the collection of parallel deduplication jobs or the one or more deduplication jobs; and
program instructions to, store, by the dynamic chunk deduplicator, the deduplication information.
18. The computer system of claim 15 , wherein the file level usage is based on analyzing access of the one or more data files wherein a range of usage thresholds are used to identify one of the deduplication chunk size to associate with the one or more data files.
19. The computer system of claim 15 , wherein the one or more deduplication actions is determined by identifying unique data segments based on hash value comparison of the collection of data segments.
20. The computer system of claim 16 , wherein one or more jobs associated with the collection of parallel deduplication jobs is based on one of the deduplication chunk size to group a respective set of the collection of data segments matching the deduplication chunk size.