Dynamic data compression selection
Aspects of dynamic data compression selection are presented. In an example method, as uncompressed data chunks of a data stream are compressed, at least one performance factor affecting selection of one of multiple compression algorithms for the uncompressed data chunks of the data stream may be determined. Each of the multiple compression algorithms may facilitate a different expected compression ratio. One of the multiple compression algorithms may be selected separately for each uncompressed data chunk of the data stream based on the at least one performance factor. Each uncompressed data chunk may be compressed using the selected one of the multiple compression algorithms for the uncompressed data chunk.
1. A method for data compression, the method comprising:
receiving uncompressed source data at a data compression system from a data storage device, the uncompressed source data comprising a plurality of uncompressed data chunks;
determining, at a first time, a first number of threads to employ to compress the uncompressed source data;
identifying, based at least in part on the first number of threads, a first compression technique from a plurality of compression techniques to use to compress one or more first uncompressed data chunks of the plurality of uncompressed data chunks;
compressing, via one or more threads, the one or more first uncompressed data chunks using the first compression technique to produce one or more first compressed data chunks, a first quantity of the one or more threads being equal to the first number of threads;
routing the one or more first compressed data chunks to a data target over a network; and
at each second time of one or more second times:
determining, at the second time, a next number of threads to employ to compress the uncompressed source data, the next number of threads being different than the first number of threads;
identifying, based at least in part on the next number of threads and utilizing one or more hardware processors of the data compression system, a next compression technique from the plurality of compression techniques to use to compress one or more next uncompressed data chunks of the plurality of uncompressed data chunks, wherein, for each of at least one second time of the one or more second times, the next compression technique is different than the first compression technique;
compressing, via at least one thread, the one or more next uncompressed data chunks of the plurality of uncompressed data chunks using the next compression technique to produce one or more next compressed data chunks, a quantity of the at least one thread being equal to the next number of threads; and
routing the one or more next compressed data chunks to the data target over the network.
2. The method for data compression as recited in claim 1 , further comprising:
estimating a first performance metric based on the first number of threads, wherein the first compression technique is identified based on the first performance metric; and
for each second time of the one or more second times, estimating a second performance metric based on the next number of threads, wherein the next compression technique is identified based on the next performance metric.
3. The method for data compression as recited in claim 1 , wherein identifying the first compression technique includes:
defining a first processor-utilization variable to be the first number of threads or a percentage of threads derived from the first number of threads; and
comparing the first processor-utilization variable to a threshold;
wherein the first compression technique includes a relatively fast compression technique when the first processor-utilization variable exceeds the threshold; and
wherein the first compression technique includes a relatively slow compression technique when the first processor-utilization variable does not exceed the threshold, the relatively fast compression technique being associated with a compression speed that is faster relative to a compression speed associated with the relatively slow compression technique.
4. The method for data compression as recited in claim 1 , wherein identifying the next compression technique includes:
selecting a compression algorithm to use to compress the one or more next uncompressed data chunks;
selecting a value or parameter with which to configure a compression algorithm to use to compress the one or more next uncompressed data chunks; or
selecting a portion of a compression algorithm to use to compress the one or more next uncompressed data chunks.
5. The method for data compression as recited in claim 1 , wherein the one or more second times include a plurality of second times, and wherein consecutive second times within the plurality of second times are separated by a defined time interval.
6. The method for data compression as recited in claim 1 , wherein the one or more second times include a plurality of second times, and wherein the plurality of second times are defined to correspond to a compression time for each of multiple sets of uncompressed data chunks in the plurality of uncompressed data chunks.
7. The method for data compression as recited in claim 1 , wherein the one or more second times include a plurality of second times, and wherein the plurality of second times are defined to correspond to a compression time for each uncompressed data chunk subsequent to the first uncompressed data chunk in the plurality of uncompressed data chunks.
8. The method for data compression as recited in claim 1 , further comprising, at each second time of one or more second times:
communicating an indication of the next compression technique used to compress the one or more next compressed data chunks to the data target.
9. The method for data compression as recited in claim 1 , wherein the first compression technique is associated with a first expected compression ratio that is different than a second expected compression ratio associated with the next compression technique identified for at least one second time of the one or more second times.
10. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:
receiving uncompressed source data from a data storage device, the uncompressed source data comprising a plurality of uncompressed data chunks;
determining, at a first time, a first number of threads to employ to compress the uncompressed source data;
identifying, based at least in part on the first number of threads, a first compression technique from a plurality of compression techniques to use to compress one or more first uncompressed data chunks of the plurality of uncompressed data chunks;
compressing, via one or more threads, the one or more first uncompressed data chunks using the first compression technique to produce one or more first compressed data chunks, a first quantity of the one or more threads being equal to the first number of threads;
routing the one or more first compressed data chunks to a data target over a network; and
at each second time of one or more second times:
determining, at the second time, a next number of threads to employ to compress the uncompressed source data, the next number of threads being different than the first number of threads;
identifying, based at least in part on the next number of threads, a next compression technique from the plurality of compression techniques to use to compress one or more next uncompressed data chunks of the plurality of uncompressed data chunks, wherein, for each of at least one second time of the one or more second times, the next compression technique is different than the first compression technique;
compressing, via at least one thread, the one or more next uncompressed data chunks of the plurality of uncompressed data chunks using the next compression technique to produce one or more next compressed data chunks, a quantity of the at least one thread being equal to the next number of threads; and
routing the one or more next compressed data chunks to the data target over the network.
11. The computer-program product as recited in claim 10 , wherein the actions further include:
estimating a first performance metric based on the first number of threads, wherein the first compression technique is identified based on the first performance metric; and
for each second time of the one or more second times, estimating a second performance metric based on the next number of threads, wherein the next compression technique is identified based on the next performance metric.
12. The computer-program product as recited in claim 10 , wherein identifying the first compression technique includes:
defining a first processor-utilization variable to be the first number of threads or a percentage of threads derived from the first number of threads; and
comparing the first processor-utilization variable to a threshold;
wherein the first compression technique includes a relatively fast compression technique when the first processor-utilization variable exceeds the threshold; and
wherein the first compression technique includes a relatively slow compression technique when the first processor-utilization variable does not exceed the threshold, the relatively fast compression technique being associated with a compression speed that is faster relative to a compression speed associated with the relatively slow compression technique.
13. The computer-program product as recited in claim 10 , wherein identifying the next compression technique includes:
selecting a compression algorithm to use to compress the one or more next uncompressed data chunks;
selecting a value or parameter with which to configure a compression algorithm to use to compress the one or more next uncompressed data chunks; or
selecting a portion of a compression algorithm to use to compress the one or more next uncompressed data chunks.
14. The computer-program product as recited in claim 10 , wherein the one or more second times include a plurality of second times, and wherein consecutive second times within the plurality of second times are separated by a defined time interval.
15. The computer-program product as recited in claim 10 , wherein the one or more second times include a plurality of second times, and wherein the plurality of second times are defined to correspond to a compression time for each of multiple sets of uncompressed data chunks in the plurality of uncompressed data chunks.
16. The computer-program product as recited in claim 10 , wherein the one or more second times include a plurality of second times, and wherein the plurality of second times are defined to correspond to a compression time for each uncompressed data chunk subsequent to the first uncompressed data chunk in the plurality of uncompressed data chunks.
17. The computer-program product as recited in claim 10 , wherein the actions further include, at each second time of one or more second times:
communicating an indication of the next compression technique used to compress the one or more next compressed data chunks to the data target.
18. The computer-program product as recited in claim 10 , wherein the first compression technique is associated with a first expected compression ratio that is different than a second expected compression ratio associated with the next compression technique identified for at least one second time of the one or more second times.
19. A system for data compression including:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which when executed on the one or more data processors, cause the one or more data processors to perform actions including:
receiving uncompressed source data from a data storage device, the uncompressed source data comprising a plurality of uncompressed data chunks;
determining, at a first time, a first number of threads to employ to compress the uncompressed source data;
identifying, based at least in part on the first number of, a first compression technique from a plurality of compression techniques to use to compress one or more first uncompressed data chunks of the plurality of uncompressed data chunks;
compressing, via one or more threads, the one or more first uncompressed data chunks using the first compression technique to produce one or more first compressed data chunks, a first quantity of the one or more threads being equal to the first number of threads;
routing the one or more first compressed data chunks to a data target over a network; and
at each second time of one or more second times:
determining, at the second time, a next number of threads to employ to compress the uncompressed source data, the next number of threads being different than the first number of threads;
identifying, based at least in part on the next number of threads, a next compression technique from the plurality of compression techniques to use to compress one or more next uncompressed data chunks of the plurality of uncompressed data chunks, wherein, for each of at least one second time of the one or more second times, the next compression technique is different than the first compression technique;
compressing, via at least one thread, the one or more next uncompressed data chunks of the plurality of uncompressed data chunks using the next compression technique to produce one or more next compressed data chunks, a quantity of the at least one thread being equal to the next number of threads; and
routing the one or more next compressed data chunks to the data target over the network.
20. The system for data compression as recited in claim 19 , wherein the actions further include:
estimating a first performance metric based on the first number of threads, wherein the first compression technique is identified based on the first performance metric; and
for each second time of the one or more second times, estimating a second performance metric based on the next number of threads, wherein the next compression technique is identified based on the next performance metric.