Probabilistic model for file-specific compression selection under SLA-constraints
One example method includes file specific compression selection. Compression metrics are generated for a chunk of a file using a reference compressor. Compression metrics for other compressors are determined from the metrics of the reference compressor. A compressor is then selected to compress the file.
1. A method, comprising:
receiving a file at a compression engine configured to select an optimal compressor from a set of compressors to compress the file;
extracting a chunk from the file;
compressing the chunk with a reference compressor included in the set of compressors;
determining compression metrics for the reference compressor;
determining compression metrics each of the compressors in the set of compressors based on the compression metrics of the reference compressor;
determining a correction factor for each of the compressors in the set of compressors;
determining estimated file metrics for each of the compressors based on the compression metrics and the correction factors; and
selecting the optimal compressor for compressing the file based on the estimated file metrics.
2. The method of claim 1 , further comprising determining the compression metrics using a model, wherein the model is linear or probabilistic.
3. The method of claim 1 , further comprising generating training data from a set of training files, the training data including a joint distributions of chunk compression ratios per chunk size, for each of the compressors relative to the reference compressor.
4. The method of claim 3 , further comprising locating the compression metrics onto the joint distribution.
5. The method of claim 4 , further comprising locating a neighborhood around the compression metrics for the chunk.
6. The method of claim 5 , further comprising computing an expected value of the correction factor within the neighborhood.
7. The method of claim 6 , further comprising applying the correction factor to the compression metrics to determine the estimated file metrics.
8. The method of claim 1 , wherein selecting the optimal compressor includes selecting the optimal compressor based on the estimated file metrics, a service level agreement, and/or an application.
9. The method of claim 1 , further comprising determining a type of the file and extracting the chunk with a corresponding module for the type of the file.
10. The method of claim 1 , wherein the chunk complies with a format of the file.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving a file at a compression engine configured to select an optimal compressor from a set of compressors to compress the file;
extracting a chunk from the file;
compressing the chunk with a reference compressor included in the set of compressors;
determining compression metrics for the reference compressor;
determining compression metrics each of the compressors in the set of compressors based on the compression metrics of the reference compressor;
determining a correction factor for each of the compressors in the set of compressors;
determining estimated file metrics for each of the compressors based on the compression metrics and the correction factors; and
selecting the optimal compressor for compressing the file based on the estimated file metrics.
12. The non-transitory storage medium of claim 11 , further comprising determining the compression metrics using a model, wherein the model is linear or probabilistic.
13. The non-transitory storage medium of claim 11 , further comprising generating training data from a set of training files, the training data including a joint distributions of chunk compression ratios per chunk size, for each of the compressors relative to the reference compressor.
14. The non-transitory storage medium of claim 13 , further comprising locating the compression metrics onto the joint distribution.
15. The non-transitory storage medium of claim 14 , further comprising locating a neighborhood around the compression metrics for the chunk.
16. The non-transitory storage medium of claim 15 , further comprising computing an expected value of the correction factor within the neighborhood.
17. The non-transitory storage medium of claim 16 , further comprising applying the correction factor to the compression metrics to determine the estimated file metrics.
18. The non-transitory storage medium of claim 11 , wherein selecting the optimal compressor includes selecting the optimal compressor based on the estimated file metrics, a service level agreement, and/or an application.
19. The non-transitory storage medium of claim 11 , further comprising determining a type of the file and extracting the chunk with a corresponding module for the type of the file.
20. The non-transitory storage medium of claim 11 , wherein the chunk complies with a format of the file.