Aligning variable sized compressed data to fixed sized storage blocks
Preparing data for deduplication including: generating, by a storage system for a compressed data block, a padded compressed data block by padding the compressed data block to conform to a fixed block size, wherein the fixed block size is greater than a size of the compressed data block; storing, in the storage system, the padded compressed data block beginning at a block boundary of a storage device in the storage system; and performing block-based deduplication on the storage system, wherein the block-based deduplication determines whether the padded compressed data block matches one or more other padded compressed data blocks stored in the storage system.
1. A method comprising:
generating a padded compressed data block by padding a previously compressed data block to conform to a fixed block size at which a storage system performs block-based deduplication, wherein the fixed block size is greater than a size of the compressed data block;
determining whether the padded compressed data block matches one or more other padded compressed data blocks stored in the storage system; and
in response to the determination:
store a reference to at least one of the one or more other padded compressed data blocks in response to the padded compressed data block matching the at least one of the one or more other padded compressed data blocks, or
apply a compression algorithm to the padded compressed data block in response to the padded compressed data block differing from the one or more other padded compressed data blocks.
2. The method of claim 1 , wherein the compressed data block is compressed using a variable-sized compression algorithm.
3. The method of claim 1 , wherein the compressed data block is padded using a repeating element.
4. The method of claim 1 , wherein the compressed data block is an encrypted compressed data block and wherein generating the padded compressed data block comprises padding the encrypted compressed data block.
5. The method of claim 1 , further comprising:
receiving, from another storage system, the compressed data block and an indication to pad the compressed block to the fixed block size.
6. The method of claim 5 , wherein performing deduplication on the storage system using the padded compressed data block comprises removing padding from the padded compressed data block.
7. An apparatus comprising:
a memory; and
a processing device, operatively coupled to the memory, configured to:
generate a padded compressed data block by padding a previously compressed data block to conform to a fixed block size at which a storage system performs block-based deduplication, wherein the fixed block size is greater than a size of the compressed data block;
determine whether the padded compressed data block matches one or more other padded compressed data blocks stored in the storage system; and
in response to the determination:
store a reference to at least one of the one or more other padded compressed data blocks in response to the padded compressed data block matching the at least one of the one or more other padded compressed data blocks, or
apply a compression algorithm to the padded compressed data block in response to the padded compressed data block differing from the one or more other padded compressed data blocks.
8. The apparatus of claim 7 , wherein the compressed data block is compressed using a variable-sized compression algorithm.
9. The apparatus of claim 7 , wherein the compressed data block is padded using a repeating element.
10. The apparatus of claim 7 , wherein the compressed data block is an encrypted compressed data block and wherein generating the padded compressed data block comprises padding the encrypted compressed data block.
11. The apparatus of claim 7 , wherein the processing device is further configured to:
receive, from another storage system, the compressed data block and an indication to pad the compressed block to the fixed block size.
12. The apparatus of claim 11 , wherein performing deduplication on the storage system using the padded compressed data block comprises removing padding from the padded compressed data block.
13. A storage system comprising:
a plurality of storage devices; and
a processing device operatively coupled to the plurality of storage devices, configured to:
generate a padded compressed data block by padding a previously compressed data block to conform to a fixed block size at which the storage system performs block-based deduplication, wherein the fixed block size is greater than a size of the compressed data block;
determine whether the padded compressed data block matches one or more other padded compressed data blocks stored in the storage system; and
in response to the determination:
store a reference to at least one of the one or more other padded compressed data blocks in response to the padded compressed data block matching the at least one of the one or more other padded compressed data blocks, or
apply a compression algorithm to the padded compressed data block in response to the padded compressed data block differing from the one or more other padded data blocks.
14. The storage system of claim 13 , wherein the compressed data block is compressed using a variable-sized compression algorithm.
15. The storage system of claim 13 , wherein the compressed data block is padded using a repeating element.
16. The storage system of claim 13 , wherein the compressed data block is an encrypted compressed data block and wherein generating the padded compressed data block comprises padding the encrypted compressed data block.
17. The storage system of claim 13 , wherein the processing device is further configured to:
receive, from another storage system, the compressed data block and an indication to pad the compressed block to the fixed block size.