Managing data reduction in storage systems using machine learning
View Patent ↗A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.
1. A method of managing data reduction in storage systems using machine learning systems, the method comprising:
calculating a value representing a data reduction assessment for a first data block in a storage system using a hash of the data block;
using the value to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block by:
determining a number of times to iteratively perform the hash computation to obtain the converged hash;
using the number of times to train the machine learning system to determine a second number of times to iteratively perform a hash computation of a hash associated with the second data block to obtain the data reduction assessment for the second data block; and
wherein assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.
2. The method of claim 1 , further comprising:
avoiding storing the second data block in the storage system based on the probability.
3. The method of claim 1 , further comprising:
training the machine learning system to assess data reduction associated with the second data block in the storage system based on a respective behavior associated with a plurality of data blocks in the storage system, wherein the plurality of data blocks comprises the first data block.
4. The method of claim 1 , further comprising:
providing the machine learning system with the second data block, and a hash associated with the second data block; and
in response, receiving, from the machine learning system, a data reduction assessment for the second data block.
5. The method of claim 4 , wherein the data reduction assessment for the second data block is a compressibility associated with the second data block.
6. The method of claim 4 , wherein the data reduction assessment for the second data block is a dedupability associated with the second data block.
7. The method of claim 6 , wherein the machine learning system identifies at least one other data block previously assessed by the machine learning system that is identical to the second data block.
8. The method of claim 1 , wherein calculating the value representing the data reduction assessment for the first data block comprises:
calculating the hash for the first data block, wherein a probability as to whether the first data block can be reduced is preserved in the hash;
iteratively performing a hash computation of the hash for the first data block until convergence; and
calculating the value to represent an entropy of the converged hash.
9. The method of claim 1 , wherein the machine learning system is a neural network.
10. A system of managing data reduction in storage systems using machine learning systems, the system comprising a processor configured to:
calculate a value representing a data reduction assessment for a first data block in a storage system using a hash of the data block;
use the value to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block by:
determining a number of times to iteratively perform the hash computation to obtain the converged hash;
using the number of times to train the machine learning system to determine a second number of times to iteratively perform a hash computation of a hash associated with the second data block to obtain the data reduction assessment for the second data block; and
wherein assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.
11. The system of claim 10 , further configured to:
avoid storing the second data block in the storage system based on the probability.
12. The system of claim 10 , further configured to:
train the machine learning system to assess data reduction associated with the second data block in the storage system based on a respective behavior associated with a plurality of data blocks in the storage system, wherein the plurality of data blocks comprises the first data block.
13. The system of claim 10 , further configured to:
provide the machine learning system with the second data block, and a hash associated with the second data block; and
in response, receive, from the machine learning system, a data reduction assessment for the second data block.
14. The system of claim 13 , wherein the data reduction assessment for the second data block is a compressibility associated with the second data block.
15. The method of claim 13 , wherein the data reduction assessment for the second data block is a dedupability associated with the second data block.
16. The system of claim 15 , wherein the machine learning system identifies at least one other data block previously assessed by the machine learning system that is identical to the second data block.
17. The system of claim 10 , wherein the processor configured to calculate the value representing the data reduction assessment for the first data block is further configured to:
calculate the hash for the first data block, wherein a probability as to whether the first data block can be reduced is preserved in the hash;
iteratively perform a hash computation of the hash for the first data block until convergence; and
calculate the value to represent an entropy of the converged hash.
18. A computer program product for managing data reduction in storage systems using machine learning systems, the computer program product comprising: a non-transitory computer readable storage medium having computer executable program code embodied therewith, the program code executable by a computer processor to:
calculate a value representing a data reduction assessment for a first data block in a storage system using a hash of the data block;
use the value to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block by:
determining a number of times to iteratively perform the hash computation to obtain the converged hash;
using the number of times to train the machine learning system to determine a second number of times to iteratively perform a hash computation of a hash associated with the second data block to obtain the data reduction assessment for the second data block; and
wherein assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.