Using machine learning to select compression algorithms for compressing binary datasets
A data model is trained to predict compressibility of binary data structures based on component entropy and predict relative compression efficiency for various compression algorithms based on component size. A recommendation engine in a storage system uses the data model to predict compressibility of binary data and determines whether to compress the binary data based on predicted compressibility. If the recommendation engine determines that compression of the binary data is justified, then a compression algorithm is recommended based on predicted relative compression efficiency. For example, the compression algorithm predicted to yield the greatest compression ratio or shortest compression/decompression time may be recommended.
1. An apparatus comprising:
at least one compute node that manages access to non-volatile storage, the compute node configured to respond to commands from host nodes to access host application data stored on the non-volatile storage, wherein the host application data comprises binary data;
a data model that has been trained to predict compression efficiency of binary data structures by a plurality of data compression algorithms based on sizes of components of the binary data structures, where each of the binary data structures comprises a header, metadata, signature, encoding, and a plurality of the components; and
a recommendation engine that uses the data model to determine which one of the plurality of data compression algorithms will most efficiently compress selected binary data and recommends that compression algorithm;
wherein the compute node compresses the selected binary data using the recommended compression algorithm.
2. The apparatus of claim 1 wherein the data model that has been trained to predict compressibility of the binary data structures based on component entropy, and wherein the recommendation engine uses the data model to determine whether the selected binary data is sufficiently compressible to justify compression.
3. The apparatus of claim 1 wherein the selected binary data is a binary data structure selected from the group consisting of: executable files, image files, audio files, video files, lists, arrays, stacks, queues, and trees.
4. The apparatus of claim 1 wherein the selected binary data is a block comprising at least a portion of a binary data structure selected from the group consisting of: executable files, image files, audio files, and video files.
5. The apparatus of claim 1 wherein the recommendation engine uses the data model to identify the recommended compression algorithm based on predicted compression ratio of the selected binary data by each of the compression algorithms.
6. The apparatus of claim 2 wherein the data model is trained to predict compressibility using statistical features comprising: mean, median, mode, standard deviation, skewness, and kurtosis of binary data structure components.
7. The apparatus of claim 1 wherein the recommendation engine provides the recommendation to the compute node.
8. A method comprising:
in a storage system comprising at least one compute node that manages access to non-volatile storage, the compute node configured to respond to commands from host nodes to access host application data stored on the non-volatile storage, wherein the host application data comprises binary data:
predicting compression efficiency of selected binary data by a plurality of compression algorithms using a data model that has been trained to predict compression efficiency of binary data structures with the data compression algorithms based on sizes of components of the binary data structures, where each of the binary data structures comprises a header, metadata, signature, encoding, and a plurality of the components; and
based on predicted compressibility, selecting one of the plurality of data compression algorithms that will most efficiently compress the selected binary data;
recommending the selected compression algorithm; and
compressing the selected binary data using the recommended compression algorithm.
9. The method of claim 8 wherein the data model that has been trained to predict compressibility of the binary data structures based on component entropy and comprising using the data model to determine whether the selected binary data is sufficiently compressible to justify compression.
10. The method of claim 8 comprising training the data model with binary data structures selected from the group consisting of: executable files, image files, audio files, and video files.
11. The method of claim 8 wherein predicting compression efficiency of the selected binary data comprising predicting compression efficiency of a block comprising at least a portion of one or more of: executable files, image files, audio files, and video files.
12. The method of claim 8 comprising using the data model to identify the recommended compression algorithm based on predicted compression ratio of the selected binary data with each of the compression algorithms.
13. The method of claim 9 comprising training the data model to predict compressibility using statistical features comprising: mean, median, mode, standard deviation, skewness, and kurtosis of binary data structure components.
14. The method of claim 8 comprising providing the recommendation to the compute node.
15. A computer-readable storage medium storing instructions that when executed by a compute node cause a storage system to perform a method for data compression, the method comprising:
predicting compression efficiency of selected binary data by a plurality of compression algorithms using a data model that has been trained to predict compression efficiency of binary data structures by the data compression algorithms based on sizes of components of the binary data structures, where each of the binary data structures comprises a header, metadata, signature, encoding, and a plurality of the components; and
based on predicted compression efficiency, selecting one of the plurality of data compression algorithms that will most efficiently compress the selected binary data;
recommending the selected compression algorithm; and
compressing the selected binary data using the recommended compression algorithm.
16. The computer-readable storage medium of claim 15 wherein the data model has been trained to predict compressibility of binary data structures based on component entropy and the method comprises using the data model to determine whether the selected binary data is sufficiently compressible to justify compression.
17. The computer-readable storage medium of claim 15 wherein the method comprises training the data mode with binary data structures selected from the group consisting of: executable files, image files, audio files, and video files.
18. The computer-readable storage medium of claim 15 wherein predicting compression efficiency of the selected binary data comprises predicting compression efficiency of a block comprising at least a portion of one or more of: executable files, image files, audio files, video files, lists, arrays, stacks, queues, and trees.
19. The computer-readable storage medium of claim 15 comprising using the data model to identify the recommended compression algorithm based on predicted compression ratio of the selected binary data by each of the compression algorithms.
20. The computer-readable storage medium of claim 16 comprising training the data model to predict compressibility using statistical features comprising: mean, median, mode, standard deviation, skewness, and kurtosis of binary data structure components.