Methods for estimating cost savings using deduplication and compression in a storage system
Methods for estimating cost savings in a storage system using an external host system. One method includes accessing over a communication network data from a unit of storage of a data storage system, wherein each of the blocks of data is uncompressed. A plurality of blocks is parsed from the data. A plurality of fingerprints is generated from the blocks using a hash algorithm. A deduplication ratio is estimated for the plurality of blocks stored in the unit of storage using a hyperloglog algorithm and a first plurality of buckets compartmentalizing the plurality of blocks, wherein the first plurality of buckets is defined by precision bits of the plurality of fingerprints. An effective compression ratio is estimated for the plurality of blocks stored in the unit of storage using the hyperloglog algorithm and a second plurality of buckets compartmentalizing the plurality of blocks, wherein the second plurality of buckets is defined by ranges of compression ratios.
1. A method for estimation, comprising:
accessing, over a communication network, data from a unit of storage of a data storage system;
parsing a plurality of blocks from the data, wherein each of the plurality of blocks of data is uncompressed;
determining a compression ratio for each of the blocks;
for each of the blocks, assigning the block to one of a plurality of buckets based on the determined compression ratio for the block, wherein the plurality of buckets are associated with respective ranges of compression ratios;
estimating a cardinality of each of the plurality of buckets using a hyperloglog technique; and
estimating a post-deduplication compression ratio for the plurality of blocks stored in the unit of storage based on the respective ranges of compression ratios associated with each of the buckets and the estimated cardinality of each of the buckets.
2. The method of claim 1 , comprising:
compressing each block of the plurality of blocks, wherein, for each of the blocks, the compression ratio for the block is determined based on a compressed size of the block.
3. The method of claim 2 , wherein each cardinality represents an estimated number of unique blocks for a respective one of the buckets.
4. The method of claim 1 , wherein estimating the post-deduplication compression ratio comprises:
for each of the buckets, determining a number of compressed blocks for the bucket by dividing the estimated cardinality of the bucket by the compression ratio associated with the bucket.
5. The method of claim 4 , wherein estimating the post-deduplication compression ratio comprises:
determining a total number of compressed blocks by adding the number of compressed blocks of each bucket.
6. The method of claim 5 , wherein estimating the post-deduplication compression ratio comprises:
determining the estimated post-deduplication compression ratio for the plurality of blocks by dividing a total number of blocks in the plurality of blocks by the total number of compressed blocks.
7. The method of claim 1 , wherein the unit of storage comprises a volume or LUN.
8. A method for estimation, comprising:
accessing, over a communication network, data from a unit of storage of a data storage system;
parsing a plurality of blocks from the data, wherein each of the plurality of blocks of data is uncompressed;
determining a compression ratio for each of the blocks;
for each of the blocks, assigning the block to one of a plurality of buckets based on the determined compression ratio for the block, wherein the plurality of buckets are associated with respective ranges of compression ratios;
generating a plurality of fingerprints based on the plurality of blocks using a hash algorithm;
estimating a cardinality of each of the plurality of buckets, each cardinality representing an estimated number of unique blocks for a respective one of the buckets; and
estimating a post-deduplication compression ratio for the plurality of blocks stored in the unit of storage based on the respective ranges of compression ratios associated with each of the buckets and the estimated cardinality of each of the buckets.
9. The method of claim 8 , comprising:
compressing each block of the plurality of blocks, wherein, for each of the blocks, the compression ratio for the block is determined based on a compressed size of the block.
10. The method of claim 8 , wherein estimating the post-deduplication compression ratio comprises:
for each of the buckets, determining a number of compressed blocks for the bucket by dividing the cardinality of the bucket by the compression ratio associated with the bucket;
determining a total number of compressed blocks by adding the number of compressed blocks of each bucket; and
determining the estimated post-deduplication compression ratio for the plurality of blocks by dividing a total number of blocks in the plurality of blocks by the total number of compressed blocks.
11. The method of claim 8 , wherein, for each of the buckets, the cardinality the bucket is estimated using a hyperloglog technique.
12. A non-transitory computer-readable medium comprising program instructions to:
access, over a communication network, data from a unit of storage of a data storage system;
parse a plurality of blocks from the data, wherein each of the plurality of blocks of data is uncompressed;
determine a compression ratio for each of the blocks;
for each of the blocks, assign the block to one of a plurality of buckets based on the determined compression ratio for the block, wherein the plurality of buckets are associated with respective ranges of compression ratios;
estimate a cardinality of each of the plurality of buckets, each cardinality representing an estimated number of unique blocks for a respective one of the buckets; and
estimate a post-deduplication compression ratio for the plurality of blocks stored in the unit of storage based on the respective ranges of compression ratios associated with each of the buckets and the estimated cardinality of each of the buckets.
13. The non-transitory computer-readable medium of claim 12 , comprising program instructions to:
for each of the buckets, determine a number of compressed blocks for the bucket by dividing the estimated cardinality of the bucket by the compression ratio associated with the bucket
determine a total number of compressed blocks by adding the number of compressed blocks of each bucket; and
determine the estimated post-deduplication compression ratio for the plurality of blocks by dividing a total number of blocks in the plurality of blocks by the total number of compressed blocks.
14. The non-transitory computer-readable medium of claim 12 , comprising program instructions to use a hyperloglog technique estimated the cardinality of each of the buckets.
15. The non-transitory computer-readable medium of claim 12 , comprising program instructions to compress each block of the plurality of blocks, wherein, for each of the blocks, the compression ratio for the block is determined based on a compressed size of the block.
16. The non-transitory computer-readable medium of claim 13 , comprising program instructions to:
determine a total number of compressed blocks by adding the number of compressed blocks of each bucket.
17. The non-transitory computer-readable medium of claim 16 , comprising program instructions to:
determine the estimated post-deduplication compression ratio for the plurality of blocks by dividing a total number of blocks in the plurality of blocks by the total number of compressed blocks.