Systems and methods for selecting data compression for storage data in a storage system
Storage systems and methods to improve space saving from data compression by providing a plurality of compression processes, and optionally, one or more parameters for controlling operation of the compression processes and selecting from the plurality of compression processes and the parameters to satisfy resource limits, such as CPU usage and memory usage. In one embodiment, the methods takes into account the content-type, such as text file or video file, and select the compression process and parameters that provide the greatest space savings for that content type while also remaining within a defined resource-usage limit.
1. A method for determining a relationship between two compression processes, comprising;
compressing, by a compression processing computing device and a first compression process, at least a portion of a storage object and, by a second compression process, at least another portion of the storage object;
calculating, by the compression processing computing device, a compression result-pair comprising a first compression ratio achieved by the first compression process and a second compression ratio achieved by the second compression process;
determining, by the compression processing computing device, a speed difference between a compression performance of the first and the second compression processes;
when the speed difference exceeds an established threshold, calculating, by the compression processing computing device, a correlation function using at least two compression result-pairs comprising the compression result-pair and another compression result-pair calculated using the first and second compression processes and another storage object;
calculating, by the compression processing computing device, a coefficient of determination to determine the goodness of the calculated correlation function; and
storing, by the compression processing computing device and in a lookup table, the correlation function, when the coefficient of determination exceeds another established threshold.
2. The method as set forth in claim 1 , wherein the lookup table includes, for each of a plurality of respective compression processes including the first and the second compression processes, one or more compression rates, one or more compression ratios, or one or more associated resource limits.
3. The method as set forth in claim 1 , wherein the storage object comprises a data file and the method further comprises identifying, by the compression processing computing device, a file type associated with the data file based at least in part on a file suffix associated with the data file or a file header within the data file.
4. The method as set forth in claim 3 , further comprising selecting, by the compression processing computing device, the first and the second compression processes based on the identified file type.
5. A non-transitory computer readable medium having stored thereon instructions for determining a relationship between two compression processes comprising executable code which when executed by a processor, causes the processor to perform steps comprising:
compressing, by a first compression process, at least a portion of a storage object and, by a second compression process, at least another portion of the storage object;
calculating a compression result-pair comprising a first compression ratio achieved by the first compression process and a second compression ratio achieved by the second compression process;
determining a speed difference between a compression performance of the first and the second compression processes;
when the speed difference exceeds an established threshold, calculating a correlation function using at least two compression result-pairs comprising the compression result-pair and another compression result-pair calculated using the first and second compression processes and another storage object,
calculating a coefficient of determination to determine the goodness of the calculated correlation function; and
storing, in a lookup table, the correlation function, when the coefficient of determination exceeds another established threshold.
6. The non-transitory computer readable medium as set forth in claim 5 , wherein the lookup table includes, for each of a plurality of respective compression processes including the first and the second compression processes, one or more compression rates, one or more compression ratios, or one or more associated resource limits.
7. The non-transitory computer readable medium as set forth in claim 5 , wherein the storage object comprises a data file, the medium further having stored thereon at least one additional instruction comprising executable code which when executed by the processor, causes the processor to perform at least one additional step comprising identifying a file type associated with the data file based at least in part on a file suffix associated with the data file or a file header within the data file.
8. The non-transitory computer readable medium as set forth in claim 7 , further having stored thereon at least one additional instruction comprising executable code which when executed by the processor, causes the processor to perform at least one additional step comprising selecting the first and the second compression processes based on the identified file type.
9. A compression processing computing device, comprising a processor and a memory coupled to the processor, wherein the processor is configured to be capable of executing programmed instructions that are stored in the memory and comprise:
compressing, by a first compression process, at least a portion of a storage object and, by a second compression process, at least another portion of the storage object;
calculating a compression result-pair comprising a first compression ratio achieved by the first compression process and a second compression ratio achieved by the second compression process;
determining a speed difference between a compression performance of the first and the second compression processes;
when the speed difference exceeds an established threshold, calculating a correlation function using at least two compression result-pairs comprising the compression result-pair and another compression result-pair calculated using the first and second compression processes and another storage object;
calculating a coefficient of determination to determine the goodness of the calculated correlation function; and
storing, in a lookup table, the correlation function, when the coefficient of determination exceeds another established threshold.
10. The device as set forth in claim 9 , wherein the lookup table includes, for each of a plurality of respective compression processes including the first and the second compression processes, one or more compression rates, one or more compression ratios, or one or more associated resource limits.
11. The device as set forth in claim 9 , wherein the storage object comprises a data file and the processor is further configured to be capable of executing at least one additional programmed instruction that is stored in the memory and comprises identifying a file type associated with the data file based at least in part on a file suffix associated with the data file or a file header within the data file.
12. The device as set forth in claim 11 , wherein the processor is further configured to be capable of executing at least one additional programmed instruction that is stored in the memory and comprises selecting the first and the second compression processes based on the identified file type.