Managing inline data compression and deduplication in storage systems
A method is used in managing inline data compression and deduplication in storage systems. A block of data from data stored in a cache of a storage system is identified based on entropy. Entropy of the block of data is compared with a first threshold value. Based on the comparison, the block of data is either deduplicated or compressed without deduplication.
1. A method for use in managing inline data compression and deduplication in storage systems, the method comprising:
identifying, based on entropy, a block of data from data stored in a cache of a storage system;
comparing entropy of the block of data with a first threshold value;
based on the comparison, either deduplicating the block of data or compressing the block of data without deduplication.
2. The method of claim 1 , wherein identifying the block of data comprises:
identifying between 4 KB and 128 KB of data to include in the block.
3. The method of claim 1 , wherein identifying the block of data comprises:
determining entropy of chunks of data stored in the cache;
including, in the block of data, chunks of data with entropy falling below a second threshold value.
4. The method of claim 3 , wherein a chunk includes 4 KB of data.
5. The method of claim 3 , wherein a chunk includes 8 KB of data.
6. The method of claim 3 , wherein comparing the entropy of the block of data with the first threshold value comprises:
determining the entropy of the block by averaging the entropy of chunks of data in the block.
7. The method of claim 1 , wherein identifying the block of data comprises:
identifying chunks of data stored in the cache within a predetermined window of time.
8. The method of claim 1 , wherein deduplicating the block of data comprises:
deduplicating the block of data in increments of 512 B.
9. The method of claim 1 , further comprising:
determining entropy of the deduplicated block of data;
comparing the entropy of the deduplicated block of data with a third threshold value;
based on the comparison, either compressing the deduplicated block of data or writing the deduplicated block of data to storage without compression.
10. The method of claim 9 , wherein determining the entropy of the deduplicated block of data comprises:
averaging the entropy of remaining data in the deduplicated block of data.
11. A system for use in managing inline data compression and deduplication in storage systems, the system comprising a processor configured to:
identify, based on entropy, a block of data from data stored in a cache of a storage system;
compare entropy of the block of data with a first threshold value;
based on the comparison, either deduplicate the block of data or compress the block of data without deduplication.
12. The system of claim 11 , wherein the processor is further configured to:
identify between 4 KB and 128 KB of data to include in the block.
13. The system of claim 11 , wherein the processor is further configured to:
determine entropy of chunks of data stored in the cache;
include, in the block of data, chunks of data with entropy falling below a second threshold value.
14. The system of claim 13 , wherein a chunk includes 4 KB of data.
15. The system of claim 13 , wherein a chunk includes 8 KB of data.
16. The system of claim 11 , wherein the processor is further configured to:
determine the entropy of the block by averaging the entropy of chunks of data in the block.
17. The system of claim 11 , wherein the processor is further configured to:
identify chunks of data stored in the cache within a predetermined window of time.
18. The system of claim 11 , wherein the processor is further configured to:
deduplicate the block of data in increments of 512 B.
19. The system of claim 11 , wherein the processor is further configured to:
determine entropy of the deduplicated block of data;
compare the entropy of the deduplicated block of data with a third threshold value;
based on the comparison, either compress the deduplicated block of data or write the deduplicated block of data to storage without compression.
20. The system of claim 19 , wherein the processor is further configured to:
average the entropy of remaining data in the deduplicated block of data.