Managing deduplication characteristics in a storage system
A method is used in managing deduplication characteristics in a storage system. Deduplication entries stored in a deduplication cache are categorized into a set of deduplication groups based on a data deduplication probability associated with the deduplication entries. A machine learning system is used to dynamically adjust deduplication characteristics associated with the set of deduplication groups based on an I/O workload associated with the storage system.
1. A method of managing deduplication characteristics in a storage system, the method comprising:
processing an I/O workload of the storage system;
categorizing deduplication entries stored in a deduplication cache into a set of deduplication groups based on a data deduplication probability associated with the deduplication entries, wherein the categorizing comprises categorizing digest keys in the set of deduplication groups according to a duplicability of data associated with the digest keys;
training a machine learning system to categorize the digest keys into the set of deduplication groups by using data resulting from the processing; and
using the machine learning system to dynamically adjust deduplication characteristics associated with the set of deduplication groups based on the I/O workload associated with the storage system, wherein the adjusting comprises dynamically adjusting the categorizations and wherein the adjusting comprises dynamically adjusting how long the digest keys remain in the deduplication cache.
2. The method of claim 1 , further comprising:
iteratively updating the deduplication characteristics associated with the set of deduplication groups as data is processed for inline deduplication in the storage system, wherein the set of deduplication groups is updated dynamically.
3. The method of claim 2 , wherein iteratively updating the deduplication characteristics associated with the set of deduplication groups as data is processed for inline deduplication in the storage system comprises:
updating the deduplication characteristics upon processing of at least one data entry for inline deduplication.
4. The method of claim 2 , wherein each deduplication group is defined by a weighted Gaussian distribution.
5. The method of claim 4 , wherein iteratively updating the deduplication characteristics associated with the set of deduplication groups as data is processed for inline deduplication in the storage system comprises:
dynamically adjusting parameters associated with the weighted Gaussian distribution.
6. The method of claim 1 , wherein each deduplication entry in the deduplication cache includes a digest associated with a portion of data and a block mapping metadata associated with the portion of data.
7. The method of claim 6 , wherein the deduplication entries are categorized into the set of deduplication groups to maximize the data deduplication in the storage system.
8. The method of claim 7 , wherein maximizing data deduplication comprises retaining the deduplication entries in the deduplication cache for a period of time based on the data deduplication probability associated with data represented by the deduplication entries, wherein a subset of the deduplication entries is retained in the deduplication cache longer as the data deduplication probability increases.
9. The method of claim 1 , wherein categorizing deduplication entries stored in the deduplication cache into the set of deduplication groups based on the data deduplication probability associated with the deduplication entries comprises:
determining a deduplication rate for a file system on the storage system based on a deduplication entry associated with the file system and a file system identifier; and
categorizing the deduplication entry into the one of the deduplication groups based on the deduplication rate.
10. A system of managing deduplication characteristics in a storage system, the system comprising a processor configured to:
process an I/O workload of the storage system;
categorize deduplication entries stored in a deduplication cache into a set of deduplication groups based on a data deduplication probability associated with the deduplication entries, wherein the categorizing comprises categorizing digest keys in the set of deduplication groups according to a duplicability of data associated with the digest keys;
train a machine learning system to categorize the digest keys into the set of deduplication groups by using data resulting from the processing; and
use the machine learning system to dynamically adjust deduplication characteristics associated with the set of deduplication groups based on the I/O workload associated with the storage system, wherein the adjusting comprises dynamically adjusting the categorizations and wherein the adjusting comprises dynamically adjusting how long the digest keys remain in the deduplication cache.
11. The system of claim 10 , further configured to:
iteratively update the deduplication characteristics associated with the set of deduplication groups as data is processed for inline deduplication in the storage system, wherein the set of deduplication groups is updated dynamically.
12. The system of claim 11 , wherein the processor configured to iteratively update the deduplication characteristics associated with the set of deduplication groups as data is processed for inline deduplication in the storage system is further configured to:
update the deduplication characteristics upon processing of at least one data entry for inline deduplication.
13. The system of claim 11 , wherein each deduplication group is defined by a weighted Gaussian distribution.
14. The system of claim 13 , wherein the processor configured to iteratively update the deduplication characteristics associated with the set of deduplication groups as data is processed for inline deduplication in the storage system is further configured to:
dynamically adjust parameters associated with the weighted Gaussian distribution.
15. The system of claim 10 , wherein each deduplication entry in the deduplication cache includes a digest associated with a portion of data and a block mapping metadata associated with the portion of data.
16. The system of claim 15 , wherein the deduplication entries are categorized into the set of deduplication groups to maximize the data deduplication in the storage system.
17. The system of claim 16 , wherein the processor configured to maximize data deduplication is further configured to retain the deduplication entries in the deduplication cache for a period of time based on the data deduplication probability associated with data represented by the deduplication entries, wherein a subset of the deduplication entries is retained in the deduplication cache longer as the data deduplication probability increases.
18. The system of claim 10 , wherein the processor configured to categorize deduplication entries stored in the deduplication cache into the set of deduplication groups based on the data deduplication probability associated with the deduplication entries is further configured to:
determine a deduplication rate for a file system on the storage system based on a deduplication entry associated with the file system and a file system identifier; and
categorize the deduplication entry into the one of the deduplication groups based on the deduplication rate.
19. A computer program product for managing deduplication characteristics in a storage system, the computer program product comprising:
a non-transitory computer readable storage medium having computer executable program code embodied therewith, the program code executable by a computer processor to:
process an I/O workload of the storage system;
categorize deduplication entries stored in a deduplication cache into a set of deduplication groups based on a data deduplication probability associated with the deduplication entries, wherein the categorizing comprises categorizing digest keys in the set of deduplication groups according to a duplicability of data associated with the digest keys;
train a machine learning system to categorize the digest keys into the set of deduplication groups by using data resulting from the processing; and
use the machine learning system to dynamically adjust deduplication characteristics associated with the set of deduplication groups based on the I/O workload associated with the storage system, wherein the adjusting comprises dynamically adjusting the categorizations and wherein the adjusting comprises dynamically adjusting how long the digest keys remain in the deduplication cache.
20. The computer program product of claim 19 , the program code further configured to:
iteratively update the deduplication characteristics associated with the set of deduplication groups as data is processed for inline deduplication in the storage system, wherein the set of deduplication groups is updated dynamically.