Managing data reduction in storage systems using machine learning
A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.
1. A method of managing data reduction in storage systems using neural networks, the method comprising:
calculating a value representing a data reduction assessment for a first data block in a storage system using a hash of the data block; and
using the value to train a neural network to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, the value comprising a number of times to compute the hash to determine a converged hash, wherein assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.
2. The method of claim 1 , further comprising:
avoiding storing the second data block in the storage system based on the probability.
3. The method of claim 1 , further comprising:
training the neural network to assess data reduction associated with the second data block in the storage system based on a respective behavior associated with a plurality of data blocks in the storage system, wherein the plurality of data blocks comprises the first data block.
4. The method of claim 1 , further comprising:
providing the neural network with the second data block, and a hash associated with the second data block; and
in response, receiving, from the neural network, a data reduction assessment for the second data block.
5. The method of claim 4 , wherein the data reduction assessment for the second data block is a compressibility associated with the second data block.
6. The method of claim 4 , wherein the data reduction assessment for the second data block is a dedupability associated with the second data block.
7. The method of claim 6 , wherein the neural network identifies at least one other data block previously assessed by the neural network that is identical to the second data block.
8. The method of claim 1 , wherein calculating the value representing the data reduction assessment for the first data block comprises:
calculating the hash for the first data block, wherein a probability as to whether the first data block can be reduced is preserved in the hash;
iteratively performing a hash computation of the hash for the first data block until convergence; and
calculating the value to represent an entropy of the converged hash.
9. The method of claim 8 , wherein using the value to train the neural network to assess data reduction associated with the second data block comprises:
determining a number of times to iteratively perform the hash computation to obtain the converged hash; and
using the number of times to train the neural network to determine a second number of times to iteratively perform a hash computation of a hash associated with the second data block to obtain the data reduction assessment for the second data block.
10. The method of claim 1 , further comprising:
storing an entropy of the converged hash instead of the hash.
11. A system of managing data reduction in storage systems using neural networks, the system comprising a processor configured to:
calculate a value representing a data reduction assessment for a first data block in a storage system using a hash of the data block; and
use the value to train a neural network to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, the value comprising a number of times to compute the hash to determine a converged hash, wherein assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.
12. The system of claim 11 , further configured to:
avoid storing the second data block in the storage system based on the probability.
13. The system of claim 11 , further configured to:
train the neural network to assess data reduction associated with the second data block in the storage system based on a respective behavior associated with a plurality of data blocks in the storage system, wherein the plurality of data blocks comprises the first data block.
14. The system of claim 11 , further configured to:
provide the neural network with the second data block, and a hash associated with the second data block; and
in response, receive, from the neural network, a data reduction assessment for the second data block.
15. The system of claim 14 , wherein the data reduction assessment for the second data block is a compressibility associated with the second data block.
16. The method of claim 14 , wherein the data reduction assessment for the second data block is a dedupability associated with the second data block.
17. The system of claim 16 , wherein the neural network identifies at least one other data block previously assessed by the neural network that is identical to the second data block.
18. The system of claim 11 , wherein the processor configured to calculate the value representing the data reduction assessment for the first data block is further configured to:
calculate the hash for the first data block, wherein a probability as to whether the first data block can be reduced is preserved in the hash;
iteratively perform a hash computation of the hash for the first data block until convergence; and
calculate the value to represent an entropy of the converged hash.
19. The system of claim 18 , wherein the processor configured to use the value to train the neural network to assess data reduction associated with the second data block is further configured to:
determine a number of times to iteratively perform the hash computation to obtain the converged hash; and
use the number of times to train the neural network to determine a second number of times to iteratively perform a hash computation of a hash associated with the second data block to obtain the data reduction assessment for the second data block.
20. A computer program product for managing data reduction in storage systems using neural networks, the computer program product comprising:
a non-transitory computer readable storage medium having computer executable program code embodied therewith, the program code executable by a computer processor to:
calculate a value representing a data reduction assessment for a first data block in a storage system using a hash of the data block; and
use the value to train a neural network to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, the value comprising a number of times to compute the hash to determine a converged hash, wherein assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.