Method for maximum data reduction combining compression with deduplication in storage arrays
One example method includes storing a user application input/output operation (IO) in a compression cache portion of a cache memory wherein the user application IO is associated with a data block, compressing the data block to produce a compressed data block, hashing the compressed data block to create a hash that uniquely corresponds to the data block, and storing the hash in a deduplication digest portion of a cache memory, wherein the deduplication digest portion includes a plurality of groups, each of which corresponds to a particular extent to which a data block is compressible. When a compressibility of the data block exceeds a threshold, the data block is retained in the compression cache portion, and when the compressibility of the data block does not exceed the threshold, the data block is flushed from the compression cache portion to disk.
1. A method, comprising:
storing a user application input/output operation (IO) in a compression cache portion of a cache memory wherein the user application IO is associated with a data block;
compressing the data block to produce a compressed data block;
hashing the compressed data block to create a hash that uniquely corresponds to the compressed data block;
storing the hash in a deduplication digest portion of the cache memory and the deduplication digest portion includes first and second groups, each of which corresponds to a particular extent to which a data block is compressible, where the first group includes hashes that correspond to data blocks that are each relatively less compressible than data blocks whose respective hashes are included in the second group, and each of the groups corresponds to a particular extent to which a data block is able to be deduplicated, and;
retaining the compressed data block in the compression cache portion when a compressibility of the compressed data block exceeds a threshold; and
flushing the compressed data block from the compression cache portion to disk when the compressibility of the compressed data block does not exceed the threshold.
2. The method as recited in claim 1 , wherein the user application IO is one of a read request, or a write request.
3. The method as recited in claim 1 , wherein a hash in the first group is flushed from the deduplication digest portion of the cache memory relatively more quickly than a hash in the second group.
4. The method as recited in claim 3 , wherein a data block whose hash is included in the first group is retained in the compression cache portion a relatively shorter time than a data block whose hash is included in the second group.
5. The method as recited in claim 1 , wherein when the compressed data block is flushed from the compression cache portion to disk, the relative amount of space in the cache memory that is available for storage of hashes increases.
6. The method as recited in claim 1 , further comprising removing the client application IO from the cache memory, and flushing the client application IO to disk.
7. The method as recited in claim 1 , wherein the cache memory comprises one or both of: Solid State Drive (SSD) storage media; and, nonvolatile memory flash backend devices.
8. The method as recited in claim 1 , further comprising performing a data deduplication process with respect to one or more data blocks whose hashes are stored in the deduplication digest portion of the cache memory.
9. The method as recited in claim 1 , wherein when the hash is removed from the deduplication digest portion of the cache memory, the relative amount of space in the cache memory that is available for storage of compressed blocks increases.
10. The method as recited in claim 1 , wherein a data block is retained in the compression cache portion until a group of ‘n’ data blocks are written to disk, wherein the group of ‘n’ data blocks includes the data block.
11. A non-transitory storage medium having stored therein instructions which are executable by one or more hardware processors to perform operations comprising:
storing a user application input/output operation (IO) in a compression cache portion of a cache memory wherein the user application IO is associated with a data block;
compressing the data block to produce a compressed data block;
hashing the compressed data block to create a hash that uniquely corresponds to the compressed data block;
storing the hash in a deduplication digest portion of the cache memory and the deduplication digest portion includes first and second groups, each of which corresponds to a particular extent to which a data block is compressible, where the first group includes hashes that correspond to data blocks that are each relatively less compressible than data blocks whose respective hashes are included in the second group, and each of the groups corresponds to a particular extent to which a data block is able to be deduplicated, and;
retaining the compressed data block in the compression cache portion when a compressibility of the compressed data block exceeds a threshold; and
flushing the compressed data block from the compression cache portion to disk when the compressibility of the compressed data block does not exceed the threshold.
12. The non-transitory storage medium as recited in claim 11 , wherein the user application IO is one of a read request, or a write request.
13. The non-transitory storage medium as recited in claim 11 , wherein a hash in the first group is flushed from the deduplication digest portion of the cache memory relatively more quickly than a hash in the second group.
14. The non-transitory storage medium as recited in claim 13 , wherein a data block whose hash is included in the first group is retained in the compression cache portion a relatively shorter time than a data block whose hash is included in the second group.
15. The non-transitory storage medium as recited in claim 11 , wherein when the compressed data block is flushed from the compression cache portion to disk, the relative amount of space in the cache memory that is available for storage of hashes increases.
16. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise removing the client application IO from the cache memory, and flushing the client application IO to disk.
17. The non-transitory storage medium as recited in claim 11 , wherein the cache memory comprises one or both of: solid state drive (SSD) storage media; and, nonvolatile memory flash backend devices.
18. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise performing a data deduplication process with respect to one or more data blocks whose hashes are stored in the deduplication digest portion of the cache memory.
19. The non-transitory storage medium as recited in claim 11 , wherein when the hash is removed from the deduplication digest portion of the cache memory, the relative amount of space in the cache memory that is available for storage of compressed blocks increases.
20. The non-transitory storage medium as recited in claim 11 , wherein a data block is retained in the compression cache portion until a group of ‘n’ data blocks are written to disk, wherein the group of ‘n’ data blocks includes the data block.