Adaptive inline compression
A technique for managing data storage includes generating entropy of blocks on a per-block basis and selectively performing inline compression on blocks based at least in part on their entropy. Entropy of a block provides a rough measure of the block's compressibility. Thus, using per-block entropy enables a storage system to steer compression decisions, e.g., whether to compress and/or how much to compress, flexibly and with high granularity, striking a balance between throughput and storage efficiency.
1. A method of managing data storage, comprising:
receiving data;
generating entropy values of respective blocks of the data;
selectively compressing the blocks of the data based on the entropy values; and
persisting the data, including persisting compressed blocks for at least some of the data,
wherein selectively compressing the blocks of the data includes:
compressing a first set of blocks having entropy values within a first range; and
skipping compression of a second set of blocks having entropy values outside the first range,
wherein compressing the first set of blocks includes:
compressing a first subset of blocks having entropy values within a first sub-range of the first range using a first compression procedure; and
compressing a second subset of blocks having entropy values within a second sub-range of the first range using a second compression procedure,
wherein blocks of the data are arranged in a queue,
wherein the method further comprises determining, based on system busyness, whether to process the data on a per-block basis or a per-batch basis,
wherein, when processing data on a per-batch basis:
generating entropy values of respective blocks of the data is performed by sampling fewer than all blocks in the queue, and
the method further comprises selectively compressing un-sampled blocks in the queue based on entropy values generated from one or more sampled blocks in the queue.
2. The method of claim 1 , wherein the first compression procedure executes faster than the second compression procedure and yields a lesser degree of data compression than the second compression procedure.
3. The method of claim 1 , wherein generating entropy values of respective blocks includes, for at least one block, generating an entropy value based on sampling bytes within the respective block, the entropy value thus reflecting sampled bytes but not un-sampled bytes within the respective block.
4. The method of claim 1 , wherein blocks of the data are arranged in a queue, and wherein the method further comprises disabling the second compression procedure in response to a length of the queue exceeding a predetermined high water mark.
5. The method of claim 1 , further comprising:
providing a set of M blocks of data;
determining a compressibility level of a first N blocks of the M blocks of data (N<M);
based on the compressibility level of the first N blocks, selecting a processing procedure as one of (i) no compression, (ii) the first compression procedure, or (iii) the second compression procedure; and
processing a remaining M-N blocks of the M blocks using the selected processing procedure.
6. The method of claim 5 , wherein the ordered set of M blocks of data are blocks contained within a flush transaction for flushing the M blocks of data from a cache.
7. The method of claim 1 , wherein persisting the data includes:
storing a compressed block with a compression header that includes metadata describing a compression procedure used to compress the compressed block; and
storing an uncompressed block without a compression header.
8. A computerized apparatus, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
receive data;
generate entropy values of respective blocks of the data;
selectively compress the blocks of the data based on the entropy values; and
persist the data, including compressed blocks for at least some of the data,
wherein the control circuitry constructed and arranged to selectively compress the blocks of the data is further constructed and arranged to:
compress a first set of blocks having entropy values within a first range; and
skip compression of a second set of blocks having entropy values outside the first range,
wherein the control circuitry constructed and arranged to compress the first set of blocks is further constructed and arranged to:
compress a first subset of blocks having entropy values within a first sub-range of the first range using a first compression procedure; and
compress a second subset of blocks having entropy values within a second sub-range of the first range using a second compression procedure,
wherein the control circuitry is further constructed and arranged to determine, based on system busyness, whether to process the data on a per-block basis or a per-batch basis, and
wherein, when processing data on a per-batch basis, the control circuitry constructed and arranged to generate entropy values of respective blocks is further constructed and arranged, for at least one block, to generate an entropy value based on sampling bytes within the respective block, the entropy value thus reflecting sampled bytes but not un-sampled bytes within the respective block.
9. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized apparatus, cause the computerized apparatus to perform a method of managing data storage, the method comprising:
receiving data;
generating entropy values of respective blocks of the data;
selectively compressing the blocks of the data based on the entropy values; and
persisting the data, including persisting compressed blocks for at least some of the data,
wherein selectively compressing the blocks of the data includes:
compressing a first set of blocks having entropy values within a first range; and
skipping compression of a second set of blocks having entropy values outside the first range,
wherein compressing the first set of blocks includes:
compressing a first subset of blocks having entropy values within a first sub-range of the first range using a first compression procedure; and
compressing a second subset of blocks having entropy values within a second sub-range of the first range using a second compression procedure;
wherein the method further comprises determining. based on system busyness, whether to process the data on a per-block basis or a per-batch basis,
wherein, when processing data on a per-batch basis, the method further comprises:
providing a set of M blocks of data;
determining a compressibility level of a first N blocks of the M blocks of data (N<M);
based on the compressibility level of the first N blocks, selecting a processing procedure as one of (i) no compression, (ii) the first compression procedure, or (iii) the second compression procedure; and
processing a remaining M-N blocks of the M blocks using the selected processing procedure.
10. The computer program product of claim 9 , wherein the first compression procedure executes faster than the second compression procedure and yields a lesser degree of data compression than the second compression procedure.
11. The computer program product of claim 9 ,
wherein blocks of the data are arranged in a queue,
wherein generating entropy values of respective blocks of the data is performed by sampling fewer than all blocks in the queue, and
wherein the method further comprises selectively compressing un-sampled blocks in the queue based on entropy values generated from one or more sampled blocks in the queue.
12. The computer program product of claim 9 , wherein generating entropy values of respective blocks includes, for at least one block, generating an entropy value based on sampling bytes within the respective block, the entropy value thus reflecting sampled bytes but not un-sampled bytes within the respective block.
13. The computer program product of claim 9 , wherein blocks of the data are arranged in a queue, and wherein the method further comprises disabling the second compression procedure in response to a length of the queue exceeding a predetermined high water mark.
14. The computer program product of claim 9 , wherein the ordered set of M blocks of data are blocks contained within a flush transaction for flushing the M blocks of data from a cache.
15. The method of claim 1 , wherein sampling fewer than all blocks in the queue further includes generating entropy values of noncontiguous blocks in the queue.
16. The method of claim 15 , wherein generating entropy values of noncontiguous blocks in the queue further includes selecting blocks in the queue at regular intervals.