Probabilistic model for file-specific compression selection under SLA-constraints
One example method includes file specific compression selection. Compression metrics are generated for a chunk of a file. Using a set of training data, the compression metrics are corrected using a correction factor to determine estimated file compression metrics. A compressor is then selected to compress the file based on at least the estimated file compression metrics.
1. A method, comprising:
receiving a file at a compression engine configured to select a compressor from among available compressors to compress the file;
extracting a chunk from the file;
compressing the chunk with compressors that are valid for a file type of the file;
determining compression metrics for each of the valid compressors;
determining a correction factor for each of the valid compressors;
determining estimated file metrics for each of the valid compressors based on the compression metrics and the correction factors; and
selecting an optimal compressor from the valid compressors based on the file metrics.
2. The method of claim 1 , further comprising selecting a chunk size for the extracted chunk.
3. The method of claim 1 , further comprising generating training data from a set of training files, the training data including a joint distribution of correction factors.
4. The method of claim 3 , further comprising locating the compression metrics onto the joint distribution of correction factors.
5. The method of claim 4 , further comprising locating a neighborhood around the compression metrics for the chunk.
6. The method of claim 5 , further comprising computing an expected value of the correction factor within the neighborhood.
7. The method of claim 6 , further comprising applying the correction factor to the compression metrics to determine the estimated file metrics.
8. The method of claim 1 , wherein selecting an optimal compressor includes selecting the optimal compressor based on the estimated file metrics, a service level agreement, and/or an application.
9. The method of claim 1 , further comprising determining a type of the file and extracting the chunk with a corresponding module for the type of the file.
10. The method of claim 1 , wherein the chunk complies with a format of the file.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving a file at a compression engine configured to select a compressor from among available compressors to compress the file;
extracting a chunk from the file;
compressing the chunk with all compressors that are valid for a file type of the file;
determining compression metrics for each of the valid compressors;
determining a correction factor for each of the valid compressors;
determining estimated file metrics for each of the valid compressors based on the compression metrics and the correction factors; and
selecting an optimal compressor from the valid compressors based on the file metrics.
12. The non-transitory storage medium of claim 11 , further comprising selecting a chunk size for the extracted chunk.
13. The non-transitory storage medium of claim 11 , further comprising generating training data from a set of training files, the training data including a joint distribution of correction factors.
14. The non-transitory storage medium of claim 13 , further comprising locating the compression metrics onto the joint distribution of correction factors.
15. The non-transitory storage medium of claim 14 , further comprising locating a neighborhood around the compression metrics for the chunk.
16. The non-transitory storage medium of claim 15 , further comprising computing an expected value of the correction factor within the neighborhood.
17. The non-transitory storage medium of claim 16 , further comprising applying the correction factor to the compression metrics to determine the estimated file metrics.
18. The non-transitory storage medium of claim 11 , wherein selecting an optimal compressor includes selecting the optimal compressor based on the estimated file metrics, a service level agreement, and/or an application.
19. The non-transitory storage medium of claim 11 , further comprising determining a type of the file and extracting the chunk with a corresponding module for the type of the file.
20. The non-transitory storage medium of claim 11 , wherein the chunk complies with a format of the file.