HASHING WITH DIFFERING HASH SIZE AND COMPRESSION SIZE
A system for hashing a data set by identifying a data set to deduplicate based on a hash block size and to compress based on a compression block size, where the hash block size is smaller than the compression block size, defining a set of data blocks within the data set based on the hash block size, generating a hash for each data block in the set of data blocks within the data set, deduplicating a data block in the data set based on a respective hash for the data block, and compressing the data set based on the compression block size.
1 . A method, comprising:
identifying a data set to deduplicate based on a hash block size and to compress based on a compression block size, wherein the hash block size is smaller than the compression block size the hash block size being selected independent of the compression block size;
defining a set of data blocks within the data set based on the hash block size;
generating a hash for each data block in the set of data blocks within the data set;
deduplicating a data block in the data set based on a respective hash for the data block; and
compressing the data set based on the compression block size.
2 . The method of claim 1 , wherein the compression block size is at least twice a size of the hash block size.
3 . The method of claim 1 , wherein the hash block size is a divisor of the compression block size.
4 . The method of claim 1 , wherein the compression block size block size is not an integer multiple of the hash block size, wherein a second data block of the set of data blocks has a unique hash block size, wherein compressing the data set based on the compression block size comprises at least one of:
refraining from compressing the second data block; or
discarding a hash that is associated with the second data block.
5 . The method of claim 1 further comprising:
receiving, via a user interface, user input indicative of the hash block size; and
setting the hash block size based on the user input.
6 . The method of claim 1 , wherein the data set is an electronic file, wherein the compression block size is equal to a size of the electronic file.
7 . The method of claim 1 , wherein the deduplicated data block is not to be compressed.
8 . A method, comprising:
identifying a data set to deduplicate based on a hash block size and to compress based on a compression block size, wherein the hash block size is smaller than the compression block size;
defining a set of data blocks within the data set based on the hash block size;
generating a first instruction to generate a hash for each data block in the set of data blocks within the data set;
generating a second instruction to deduplicate a data block in the data set based on a respective hash for the data block to generate a deduplicated data set; and
generating a third instruction to compress the deduplicated data set based on the compression block size.
9 . The method of claim 8 , wherein the compression block size is at least twice a size of the hash block size.
10 . The method of claim 8 , wherein the hash block size is a divisor of the compression block size.
11 . The method of claim 8 , wherein the compression block size block size is not an integer multiple of the hash block size, wherein a second data block of the set of data blocks has a unique hash block size, wherein the third instruction to compress the data set based on the compression block size comprises a fourth instruction to perform at least one of:
refraining from compressing the second data block; or
discarding a hash that is associated with the second data block.
12 . The method of claim 8 further comprising:
receiving, via a user interface, user input indicative of the hash block size; and
setting the hash block size based on the user input.
13 . The method of claim 8 , wherein the data set is an electronic file, wherein the compression block size is equal to a size of the electronic file.
14 . The method of claim 8 , wherein the deduplicated data block is not to be compressed.
15 . A system, comprising:
a memory; and
a processing unit coupled to the memory, the processing unit being configured to cause the system to perform operations comprising:
identify a data set to deduplicate based on a hash block size, wherein the hash block size is smaller than a compression block size that is associated with the data set;
define a set of data blocks within the data set based on the hash block size the set of data blocks being a subset of the data set;
generate a hash for each data block in the set of data blocks within the data set; and
deduplicate a data block in the set of data blocks based on a respective hash for the data block.
16 . The system of claim 15 , wherein the compression block size is at least twice a size of the hash block size.
17 . The system of claim 15 , wherein the hash block size is a divisor of the compression block size.
18 . The system of claim 15 , wherein the compression block size block size is not an integer multiple of the hash block size, wherein a second data block of the set of data blocks has a unique hash block size.
19 . The system of claim 15 , the operations further comprising:
receive, via a user interface, user input indicative of the hash block size; and
set the hash block size based on the user input.
20 . The system of claim 15 , wherein the data set is an electronic file, wherein the compression block size is equal to a size of the electronic file.