Multiple overlapping hashes at variable offset in a hardware offload
A hardware offload includes a hash engine that performs hashing for a block-based storage system. The hash engine calculates multiple hash values for each input buffer provided by the storage system. The hash values may be calculated with variably offset and overlapping portions of the input buffer, wherein each portion is larger than the native block size of the storage system. The hardware offload may also include a compression engine that performs compression on the input buffer using the entire input buffer and/or chunks as compression domains.
1. A system comprising:
a SAN (storage area network) node that maintains a data set, the SAN node organizing the data set into input buffers that are each larger in size than a native block of the SAN node; and
a hardware offload that processes individual ones of the input buffers by calculating multiple hash values and returning the multiple hash values to the SAN node,
wherein the SAN node uses the multiple hash values to perform deduplication on the data set.
2. The system of claim 1 wherein the hash values are calculated for chunks of the input buffer, wherein the chunks have variable offsets and overlap, and each chunk is larger, equal, or smaller in size than the native block of the SAN node.
3. A method comprising:
a block-based storage system sending a single input buffer of data of a data set to a hardware offload;
the hardware offload:
calculating multiple hash values from the single input buffer received from the block-based storage system; and
returning the multiple hash values to the block-based storage system; and
the block-based storage system using the multiple hash values to perform deduplication on the data set.
4. The method of claim 3 comprising creating the input buffer with a larger size than a native block of the block-based storage system.
5. The method of claim 4 comprising the hardware offload organizing the input buffer into multiple logical chunks and hashing each chunk to generate the multiple hash values.
6. The method of claim 5 comprising forming the chunks with variable offsets.
7. The method of claim 5 comprising forming the chunks with overlap.
8. The method of claim 5 comprising forming each chunk with a larger, equal, or smaller size than the native block.
9. The method of claim 3 comprising the hardware offload compressing the input buffer as a single compression domain to generate a compressed buffer and returning the compressed buffer to the block-based storage system.
10. The method of claim 5 comprising the hardware offload compressing each chunk as a separate compression domain to generate compressed buffers and returning the compressed buffers to the block-based storage system.
11. The method of claim 5 comprising the hardware offload compressing each chunk of a selected subset of chunks as a separate compression domain to generate compressed buffers and returning the compressed buffers to the block-based storage system.
12. An apparatus comprising:
a hardware offload that performs hashing for a block-based storage system, the hardware offload comprising a hash engine that calculates multiple hash values from a single input buffer received from the block-based storage system and returns the multiple hash values to the block-based storage system, wherein the block-based storage system is configured to perform data reduction using the multiple hash values to identify data duplication.
13. The apparatus of claim 12 wherein the input buffer is larger in size than a native block of the block-based storage system.
14. The apparatus of claim 13 wherein the hash engine organizes the input buffer into multiple logical chunks and hashes each chunk.
15. The apparatus of claim 14 wherein the chunks have variable offsets.
16. The apparatus of claim 14 wherein the chunks overlap.
17. The apparatus of claim 14 wherein each chunk is larger, equal, or smaller in size than the native block.
18. The apparatus of claim 12 comprising a compression engine that compresses the input buffer as a single compression domain.
19. The apparatus of claim 14 comprising a compression engine that compresses each chunk as a separate compression domain.
20. The apparatus of claim 14 comprising a compression engine that compresses a selected subset of the chunks, each as a separate compression domain.