Hashing for deduplication through skipping selected data
A system for calculating a fingerprint across a data set by identifying a data set to hash, the data set comprising a set of data blocks, identifying data within the data set to skip, generating, by a hash engine, a hash for each data block in the set of data blocks within the data set except for the data within the data set to skip, and compressing the data.
1. A method, comprising:
identifying a data set to hash, the data set comprising a set of data blocks;
identifying data within the data set to skip;
generating, by a hash engine, a hash for each data block in the set of data blocks within the data set except for the data within the data set to skip, wherein generating, the hash for each data block in the set of data blocks within the data set except for the data within the data set to skip includes skipping a number of bytes equal to a particular byte count; and
compressing the data.
2. The method of claim 1 , wherein identifying data within the data set to skip includes determining that the data includes a repeating field.
3. The method of claim 2 , wherein the repeating field has a particular byte count, wherein generating the hash for each data block in the set of data blocks within the data set except for the data within the data set to skip includes identifying a first byte of the repeating field and skipping a number of bytes equal to the particular byte count.
4. The method of claim 1 further comprising:
providing the skipped data to a record for storage; and
providing the record of the skipped data via a user interface.
5. The method of claim 1 , wherein the data within the data set to skip includes at least one of: data that is characterized as a pattern, non-volatile memory express (NVMe) protection data, or error recovery control data.
6. The method of claim 1 , wherein identifying data within the data set to skip includes:
identifying a skip candidate in the data set;
providing the skip candidate to be provided via interface; and
receiving a confirmation of the skip candidate.
7. A system, comprising:
a memory; and
a processing unit coupled to the memory, the processing unit being configured to cause the system to perform operations comprising:
identify a data set to hash, the data set comprising a set of data blocks;
identify data within the data set to skip; and
generate, by a hash engine, a hash for each data block in the set of data blocks within the data set except for the data within the data set to skip, wherein generating the hash for each data block in the set of data blocks within the data set except for the data within the data set to skip includes skipping a number of bytes equal to a particular byte count; and
compress the data.
8. The system of claim 7 , wherein identifying data within the data set to skip includes determining that the data includes a repeating field, wherein the repeating field has the particular byte count.
9. The system of claim 7 , the operations further comprising:
providing the skipped data to a record for storage; and
providing the record of the skipped data via a user interface.
10. The system of claim 7 , wherein the data within the data set to skip includes at least one of: data that is characterized as a pattern, a header, a footer, non-volatile memory express (NVMe) protection data, or error recovery control data.
11. The system of claim 7 , wherein identifying data within the data set to skip includes:
identifying a skip candidate in the data set;
providing the skip candidate to be provided via interface; and
receiving a confirmation of the skip candidate.
12. A system, comprising:
a memory; and
one or more processors coupled to the memory, the one or more processors being configured to execute operations comprising:
identify a data set to hash, the data set comprising a set of data blocks;
identify data within the data set to skip;
generate a hash for each data block in the set of data blocks within the data set except for the data within the data set to skip, wherein generating the hash for each data block in the set of data blocks within the data set except for the data within the data set to skip includes skipping a number of bytes equal to a particular byte count; and
compress the data.
13. The system of claim 12 , wherein identifying data within the data set to skip includes determining that the data includes a repeating field.
14. The system of claim 13 , wherein the repeating field has a particular byte count, wherein generating the hash for each data block in the set of data blocks within the data set except for the data within the data set to skip includes identifying a first byte of the repeating field and skipping a number of bytes equal to the particular byte count.
15. The system of claim 12 , wherein generating the hash for each data block in the set of data blocks within the data set except for the data within the data set to skip includes skipping data based on an offset and a skip count.
16. The system of claim 12 , wherein the data within the data set to skip includes at least one of: data that is characterized as a pattern, a header, a footer, non-volatile memory express (NVMe) protection data, or error recovery control data.
17. The system of claim 12 , wherein identifying data within the data set to skip includes:
identifying a skip candidate in the data set;
providing the skip candidate to be provided via interface; and
receiving a confirmation of the skip candidate.
18. The system of claim 15 , wherein the offset is an adjustable parameter.