Optimizing data block size for deduplication
Provided herein is technology relating to data deduplication and particularly, but not exclusively, to methods and systems for determining an efficiently optimal size of data blocks to use for backing up a data source. Also provided herein are systems for identifying duplicate data in data backup applications.
1. A method comprising:
obtaining a file of a file type;
determining a deduplication block size based on the file type, wherein the deduplication block size provides a maximum compression ratio for the file type; and
deduplicating a set of data blocks of the file based on the deduplication block size and a sliding window, wherein the sliding window moves backwards responsive to not detecting a duplicate data block.
2. The method of claim 1 , wherein deduplicating the set of data blocks comprises:
determining whether a first data block of the file is identical to a previous data block.
3. The method of claim 2 , wherein determining whether the first data block is a duplicate data block comprises:
calculating a first hash value of the first data block to identify the first data block as a potential duplicate data block; and
calculating a second hash value of the first data block to confirm that the first data block is the duplicate data block.
4. The method of claim 2 , wherein deduplicating the set of data blocks further comprises:
setting a beginning of the sliding window to an end of the first data block responsive to determining that the first data block is identical to the previous data block.
5. The method of claim 4 , wherein deduplicating the set of data blocks further comprises:
replacing the duplicate data block with a pointer to the previous data block, wherein the duplicate data block is identical to the previous data block.
6. The method of claim 1 , wherein deduplicating the set of data blocks comprises:
moving the sliding window backwards by one byte responsive to not detecting the duplicate data block.
7. The method of claim 1 , wherein a size of the sliding window is the same as the deduplication block size.
8. The method of claim 1 , wherein determining the deduplication block size comprises:
determining a plurality of compression ratios for a test file of the file type, wherein each compression ratio is associated with a test block size;
determining a maximum compression ratio from the plurality of compression ratios; and
selecting a first block size associated with the maximum compression ratio as the deduplication block size.
9. An apparatus, comprising:
a memory configured to store a table associating a plurality of file types with a plurality of deduplication block sizes; and
a processor configured to:
obtain a file of a file type;
determine a deduplication block size based on the file type, wherein the deduplication block size provides a maximum compression ratio for the file type; and
deduplicate a set of data blocks of the file based on the deduplication block size and a sliding window, wherein the sliding window moves backwards responsive to not detecting a duplicate data block.
10. The apparatus of claim 9 , wherein the processor is configured to deduplicate the set of data blocks by:
determining whether a first data block of the file is identical to a previous data block.
11. The apparatus of claim 10 , wherein the processor is configured to determine whether the first data block is a duplicate data block by:
calculating a first hash value of the first data block to identify the first data block as a potential duplicate data block; and
calculating a second hash value of the first data block to confirm that the first data block is the duplicate data block.
12. The apparatus of claim 10 , wherein the processor is further configured to deduplicate the set of data blocks by:
setting a beginning of the sliding window to an end of the first data block responsive to determining that the first data block is identical to the previous data block.
13. The apparatus of claim 12 , wherein the processor is further configured to deduplicate the set of data blocks by:
replacing the duplicate data block with a pointer to the previous data block, wherein the duplicate data block is identical to the previous data block.
14. The apparatus of claim 9 , wherein the processor is configured to deduplicate the set of data blocks by:
moving the sliding window backwards by one byte responsive to not detecting the duplicate data block.
15. The apparatus of claim 9 , wherein a size of the sliding window is the same as the deduplication block size.
16. The apparatus of claim 9 , wherein determining the deduplication block size comprises:
determining a plurality of compression ratios for a test file of the file type, wherein each compression ratio is associated with a test block size;
determining a maximum compression ratio from the plurality of compression ratios; and
selecting a first block size associated with the maximum compression ratio as the deduplication block size.
17. An apparatus, comprising:
means for determining a deduplication block size based on a file type, wherein the deduplication block size provides a maximum compression ratio for the file type; and
means for deduplicating a set of data blocks of a file based on the determined deduplication block size and a sliding window, wherein the sliding window moves backwards responsive to not detecting a duplicate data block.
18. The apparatus of claim 17 , wherein deduplicating the set of data blocks comprises:
determining whether a first data block of the file is identical to a previous data block.
19. The apparatus of claim 18 , wherein deduplicating the set of data blocks further comprises:
setting a beginning of the sliding window to an end of the first data block responsive to determining that the first data block is identical to the previous data block.
20. The apparatus of claim 17 , wherein deduplicating the set of data blocks comprises:
moving the sliding window backwards by one byte responsive to not detecting the duplicate data block.