Data reduction in block-based storage systems using content-based block alignment
A method of data reduction in a block-based data storage system includes selecting a starting position in a block based on a deterministic function of block data content. Then for an unaligned block beginning at the selected starting position, a block digest (e.g., block hash) is generated and compared with stored block digests of stored data blocks. If there is a match, and the stored block matches the unaligned block, then a reference to the stored block is stored in place of the unaligned block, and otherwise the unaligned block and a corresponding digest are stored. The storing of references to already stored blocks, without the constraint of observing aligned-block boundaries, realizes increased savings of physical storage space.
1. A method of data reduction in a fixed block-based data storage system, comprising:
selecting a starting position based on a deterministic function of block data content of an aligned block; and
performing iterations of a loop including (1) for an unaligned block beginning at the starting position of the aligned block, generating a fixed size block digest and comparing the generated block digest with stored block digests of corresponding stored blocks in a block store, (2) when the comparing results in a match, and the corresponding stored block is determined to match the unaligned block, then (a) storing a reference to the stored block in place of the unaligned block, and (b) proceeding to a next iteration of the loop using the selected starting position in a next aligned block, and (3) when the comparing does not result in a match, (a) storing the unaligned block and a corresponding block digest and offset in the block store, and (b) repeating the above steps beginning with the selecting step to select a new starting position in a next aligned block.
2. The method of claim 1 , wherein selecting a starting position includes performing a repeated sliding window calculation over successive segments of the aligned block, and making an exit decision based on the sliding window calculation satisfying an exit criterion.
3. The method of claim 2 , wherein the sliding window calculation uses a function that produces a semi-random number for each window.
4. The method of claim 3 , wherein the sliding window has a window size is in a range from 4 to 8 bytes.
5. The method of claim 2 , wherein the sliding window calculations progress through successive windows with a step size substantially less than a size of the aligned block, and the window size is greater than the step size.
6. The method of claim 5 , wherein the step size is one byte.
7. The method of claim 2 , wherein the exit criterion is statistically well distributed with respect to a block size of the aligned block, promoting calculation of one hash per aligned block.
8. The method of claim 7 , wherein the exit criterion is a match between a sliding window calculation result and a predefined fixed value.
9. A data storage system, comprising:
physical data storage devices for storing a block store of blocks of user data;
interface circuitry providing respective interfaces to the physical storage devices and to host computers generating storage commands received and executed by the data storage system; and
processing circuitry configured and operative to store and execute computer program instructions to perform a method of data reduction including:
selecting a starting position based on a deterministic function of block data content of an aligned block; and
performing iterations of a loop including (1) for an unaligned block beginning at the starting position of the aligned block, generating a fixed size block digest and comparing the generated block digest with stored block digests of corresponding stored blocks in the block store, (2) when the comparing results in a match, and the corresponding stored block is determined to match the unaligned block, then (a) storing a reference to the stored block in place of the unaligned block, and (b) proceeding to a next iteration of the loop using the selected starting position in a next aligned block, and (3) when the comparing does not result in a match, (a) storing the unaligned block and a corresponding block digest and offset in the block store, and (b) repeating the above steps beginning with the selecting step to select a new starting position in the next aligned block.
10. The data storage system of claim 9 , wherein selecting a starting position includes performing a repeated sliding window calculation over successive segments of the aligned block, and making an exit decision based on the sliding window calculation satisfying an exit criterion.
11. The data storage system of claim 10 , wherein the sliding window calculation uses a function that produces a semi-random number for each window.
12. The data storage system of claim 11 , wherein the sliding window has a window size is in a range from 4 to 8 bytes.
13. The data storage system of claim 10 , wherein the sliding window calculations progress through successive windows with a step size substantially less than a size of the aligned block, and the window size is greater than the step size.
14. The data storage system of claim 13 , wherein the step size is one byte.
15. The data storage system of claim 10 , wherein the exit criterion is statistically well distributed with respect to a block size of the aligned block, promoting the calculation of one hash per aligned block.
16. The data storage system of claim 15 , wherein the exit criterion is a match between a sliding window calculation result and a predefined fixed value.