Selectively performing inline compression based on data entropy
A technique for managing data storage obtains a batch of chunks of data. The technique generates, using multiple pipelined instructions operating on the batch, a measure of data entropy for each of the chunks in the batch. The technique selectively compresses chunks in the batch based at least in part on the measures of data entropy generated for the respective chunks.
1. A method of managing data storage in a computerized system, the method comprising:
obtaining a batch of chunks, the batch including a plurality of chunks of data;
generating, using multiple pipelined instructions operating on the batch, a measure of data entropy for each of the plurality of chunks in the batch; and
selectively compressing chunks of the plurality of chunks based at least in part on the measures of data entropy generated for the respective chunks,
wherein the multiple pipelined instructions include pipelined processor instructions of a single CPU (central processing unit), and wherein the pipelined processor instructions include N load instructions from a cache, wherein N is at least as large as a latency in accessing the cache by the CPU as measured in clock cycles of the CPU.
2. The method of claim 1 , wherein the chunks within the batch are of equal size.
3. The method of claim 1 , wherein pipelined instructions include load instructions for loading respective data elements from a cache, and wherein a number of the plurality of chunks is at least as great as a number of clock cycles required for the CPU to load each data element from the cache.
4. The method of claim 3 , further comprising executing the load instructions sequentially during respective clock cycles of the CPU.
5. The method of claim 3 , wherein the data elements represent counts of symbols occurring in the chunks.
6. The method of claim 5 , wherein the data elements include a plurality of data elements for each chunk.
7. The method of claim 6 , wherein the plurality of data elements for each chunk includes a data element for each symbol allowed to occur in the chunk.
8. The method of claim 3 , wherein each of the data elements has a respective address location in the cache which is not shared with any other of the data elements.
9. The method of claim 1 , further comprising providing a look up table (LUT) that associates (i) counts of symbol occurrences with (ii) entropy components corresponding to the respective counts.
10. The method of claim 9 , wherein the entropy components in the LUT are integer values.
11. The method of claim 10 , wherein generating the measure of data entropy for a chunk includes (i) obtaining a set of entropy components from the LUT based on counts of symbols appearing in the chunk and (ii) generating a sum of the obtained entropy components.
12. The method of claim 11 , wherein generating the sum of entropy components includes adding entropy components across all unique symbols in the chunk.
13. The method of claim 11 , wherein generating the sum of entropy components includes adding entropy components across fewer than all unique symbols in the chunk.
14. The method of claim 13 , wherein the symbols have a numerical sequence, and wherein adding entropy components across fewer than all unique symbols in the chunk includes adding entropy components from every Nth symbol in the sequence, where N is an integer greater than one, and excluding from the sum other symbols in the sequence.
15. A computerized system, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
obtain a batch of chunks, the batch including a plurality of chunks of data;
generate, using multiple pipelined instructions operating on the batch, a measure of data entropy for each of the plurality of chunks in the batch; and
selectively compress chunks of the plurality of chunks based at least in part on the measures of data entropy generated for the respective chunks,
wherein the multiple pipelined instructions include pipelined processor instructions of a single CPU (central processing unit), and wherein the pipelined processor instructions include N load instructions from a cache, wherein N is at least as large as a latency in accessing the cache by the CPU as measured in clock cycles of the CPU.
16. The system of claim 15 , wherein the chunks within the batch are of equal size.
17. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a computerized system, cause the control circuitry to perform a method of managing data storage, the method comprising:
obtaining a batch of chunks, the batch including a plurality of chunks of data;
generating, using multiple pipelined instructions operating on the batch, a measure of data entropy for each of the plurality of chunks in the batch; and
selectively compressing chunks of the plurality of chunks based at least in part on the measures of data entropy generated for the respective chunks,
wherein the multiple pipelined instructions include pipelined processor instructions of a single CPU (central processing unit), and wherein the pipelined processor instructions include N load instructions from a cache, wherein N is at least as large as a latency in accessing the cache by the CPU as measured in clock cycles of the CPU.
18. The method of claim 1 , wherein the CPU avoids inactivity while waiting for results of a first load instruction to arrive from the cache by performing successive load instructions of the pipelined instructions.
19. The computerized system of claim 15 , wherein the CPU avoids inactivity while waiting for results of a first load instruction to arrive from the cache by performing successive load instructions of the pipelined instructions.
20. The computerized system of claim 15 , wherein the control circuitry is further constructed and arranged to provide a look up table (LUT) that associates (i) counts of symbol occurrences with (ii) entropy components corresponding to the respective counts.
21. The computer program product of claim 17 , wherein the CPU avoids inactivity while waiting for results of a first load instruction to arrive from the cache by performing successive load instructions of the pipelined instructions.
22. The computer program product of claim 17 , wherein the method further comprises providing a look up table (LUT) that associates (i) counts of symbol occurrences with (ii) entropy components corresponding to the respective counts.