Statistical and neural network approach for data characterization to reduce storage space requirements
A data model is trained to determine whether data is raw, compressed, and/or encrypted. The data model may also be trained to recognize which compression algorithm was used to compress data and predict compression ratios for the data using different compression algorithms. A storage system uses the data model to independently identify raw data. The raw data is grouped based on similarity of statistical features and group members are compressed with the same compression algorithm and may be encrypted after compression with the same encryption algorithm. The data model may also be used to identify sub-optimally compressed data, which may be uncompressed and grouped for compression using a different compression algorithm.
1. A method comprising:
in a storage system comprising at least one compute node that manages access to non-volatile storage, the compute node configured to respond to commands from host nodes to store host application data including a first portion of host application data and a second portion of host application data received from the host nodes without notification of compression state on the non-volatile storage:
using a machine learning model that has been trained to recognize, from compressed data alone, data compression state, ones of a plurality of different compression algorithms used to perform compression on data, and compressibility of different data types with the ones of the plurality of different compression algorithms:
calculating, using only the first portion of host application data and the machine learning model, that the first portion of host application data is in an uncompressed state and, in response, prompting compression of the first portion of host application data;
calculating, using only the second portion of host application data and the machine learning model, that the second portion of host application data is in a compressed state and, in response:
calculating, using only the second portion of host application data and the machine learning model, that a first compression algorithm of the plurality of different compression algorithms with which the machine learning model was trained was used by the host nodes to compress the second portion of host application data;
determining, using only the second portion of host application data and the machine learning model, that a second compression algorithm of the plurality of different compression algorithms with which the machine learning model was trained is likely to yield the greatest compression ratio in comparison with other compression algorithms with which the machine learning model has been trained, including the first compression algorithm; and
prompting use of the second compression algorithm rather than the first compression algorithm with the second portion of host application data.
2. The method of claim 1 comprising grouping data structures or blocks of the host application data based on similarity of statistical features.
3. The method of claim 2 comprising compressing all data structures or blocks in a group with a single compression algorithm.
4. The method of claim 3 wherein the machine learning model has been trained to recognize encryption state and comprising using the machine learning model to identify the encryption state of the host application data stored on the non-volatile storage and grouping uncompressed and unencrypted data structures or blocks of the host application data based on similarity of statistical features and prompting compression and encryption of all the data structures or blocks in the group with the single compression algorithm and a single encryption algorithm.
5. The method of claim 1 comprising using the machine learning model to identify a different compression algorithm that projects to yield a better compression ratio.
6. The method of claim 1 comprising uncompressing and grouping a data structure or block in response to identifying a different compression algorithm that projects to yield a better compression ratio.
7. The method of claim 1 comprising training the machine learning model to recognize compression state using statistical features.
8. The method of claim 1 comprising training the machine learning model to recognize compression state using one or more of entropy, Chi square test, Pearson's correlation coefficient, mean, median, mode, standard deviation, skewness, and kurtosis.
9. An apparatus comprising:
at least one compute node that manages access to non-volatile storage, the compute node configured to respond to commands from host nodes to store host application data including a first portion of host application data and a second portion of host application data received from the host nodes without notification of compression state on the non-volatile storage;
a machine learning data model that has been trained to recognize from compressed data alone:
data compression state;
ones of a plurality of different compression algorithms used to perform compression on data; and
compressibility of different data types with the ones of the plurality of different compression algorithms; and
a recommendation engine that uses the machine learning data model to analyze host application data stored on the non-volatile storage, including the first portion of host application data and the second portion of host application data, to:
calculate, using only the first portion of host application data and the machine learning model, that the first portion of host application data is in an uncompressed state and, in response, prompt compression of the first portion of host application data;
calculate, using only the second portion of host application data and the machine learning model, that the second portion of host application data is in a compressed state and, in response:
calculate, using only the second portion of host application data and the machine learning model, that a first compression algorithm of the plurality of different compression algorithms with which the machine learning model was trained was used by the host nodes to compress the second portion of host application data;
determine, using only the second portion of host application data and the machine learning model, that a second compression algorithm of the plurality of different compression algorithms with which the machine learning model was trained is likely to yield the greatest compression ratio in comparison with other compression algorithms with which the machine learning model has been trained, including the first compression algorithm; and
prompt use of the second compression algorithm rather than the first compression algorithm with the second portion of host application data.
10. The apparatus of claim 9 wherein the recommendation engine groups data structures or blocks of the host application data based on similarity of statistical features.
11. The apparatus of claim 10 wherein the recommendation engine prompts compression of all data structures or blocks in a group with a single compression algorithm.
12. The apparatus of claim 11 wherein the machine learning model has been trained to recognize encryption state and the recommendation engine uses the machine learning model to identify the encryption state of the host application data stored on the non-volatile storage and groups uncompressed and unencrypted data structures or blocks of the host application data based on similarity of statistical features and prompts compression and encryption of all the data structures or blocks in the group with the single compression algorithm and a single encryption algorithm.
13. The apparatus of claim 9 wherein the recommendation engine uses the machine learning model to identify a different compression algorithm that projects to yield a better compression ratio.
14. The apparatus of claim 9 wherein the recommendation engine prompts uncompressing and grouping of a data structure or block in response to identifying a different compression algorithm that projects to yield a better compression ratio.
15. The apparatus of claim 9 wherein the machine learning model is trained to recognize compression state using statistical features.
16. The apparatus of claim 15 wherein the statistical features comprise one or more of entropy, Chi square test, Pearson's correlation coefficient, mean, median, mode, standard deviation, skewness, and kurtosis.
17. The apparatus of claim 16 wherein the statistical features further comprise data structure type and data structure size.
18. The apparatus of claim 17 wherein the data structure comprises at least one of files, lists, arrays, stacks, queues, and trees.