System and method for computing message digests
A data de-duplication approach leverages acceleration hardware in SSDs for performing digest computations used in de-duplication operations and support on behalf of an attached host, thereby relieving the host from the computing burden of the digest computation in de-duplication (de-dupe) processing. De-dupe processing typically involve computation and comparison of message digests (MD) and/or hash functions. Such MD functions are often also employed for cryptographic operations such as encryption and authentication. Often, SSDs include onboard hardware accelerators for MD functions associated with security features of the SSDs. However, the hardware accelerators may also be invoked for computing a message digest result and returning the result to the host, effectively offloading the burden of MD computation from the host, similar to an external hardware accelerator, but without redirecting the data since the digest computation is performed on a data stream passing through the SSD for storage.
1. A method comprising:
receiving a first write request to store a first block of data at a storage device coupled with a host;
implementing noninvasive snooping of the first block of data as the first block of data is received at the storage device;
computing, via use of a hardware accelerator at the storage device, a first message digest of the first block of data responsive to receiving a first command to compute the first message digest in conjunction with fulfilling the first write request, the first message digest computed via a stateless, flow-through computation applied to the first block of data as the first block of data passes on a data path from the host to the storage device;
storing the first block of data in a storage medium at the storage device;
receiving a second write request to store a second block of data at the storage device;
implementing noninvasive snooping of the second block of data as the second block of data is received at the storage device;
computing, via use of the hardware accelerator at the storage device, a second message digest of the second block of data responsive to receiving a second command to compute the second message digest in conjunction with fulfilling the second write request, the second message digest computed via a stateless, flow-through computation applied to the second block of data as the second block of data passes on the data path from the host to the storage device;
storing the second block of data in the storage medium at the storage device; and
sending the first and second message digests to the host to enable the host to compare the first message digest with the second message digest for detecting duplication of blocks of data.
2. The method of claim 1 , comprising computing the first and second message digests without redirecting respective first and second blocks of data from a data path to the storage medium.
3. The method of claim 1 , computing the first and second message digests comprises the hardware accelerator separately applying a digest function to the first and second blocks of data as the first and second blocks of data pass on a data path from the host to the storage device.
4. The method of claim 1 , comprising the hardware accelerator to also perform compression and security functions for data passing between the host and the storage device, the security functions including encryption, decryption or secure hash computation.
5. The method of claim 1 , further comprising:
identifying a third block of data previously written to the storage medium;
retrieving the third block of data and store to a buffer for offline message digest computation;
compute, via use of the hardware accelerator at the storage device, a third message digest of the third block of data stored to the buffer; and
sending the third message digest to the host to enable the host to compare the third message digest with one or more other message digests previously sent from the storage device to detect duplication of blocks of data stored at the storage device.
6. The method of claim 1 , the storage device comprising a solid state drive and the storage medium includes NAND memory.
7. A storage device comprising:
an interface coupled with a host device;
a storage medium; and
a hardware accelerator to compute message digests for respective blocks of data received from the host device, the hardware accelerator to implement noninvasive snooping of the respective blocks of data to be stored at the storage medium, the noninvasive snooping to occur as the respective blocks of data are received at the storage medium from the host device, the computed message digests are computed by the hardware accelerator via a stateless, flow-through computation applied to the respective blocks of data as the respective blocks of data pass from the host device to the storage medium, the respective blocks of data to traverse a data path from the host device to the storage medium, the hardware accelerator to compute the message digests responsive to separate commands from the host device for respective write requests to store the respective blocks of data in the storage medium, the message digests computed in conjunction with fulfilling respective write requests, computed message digests sent to the host device through the interface, the host device to use the computed message digests to detect duplication of blocks of data stored in the storage medium.
8. The storage device of claim 7 , comprising the hardware accelerator to compute message digests without redirection of the respective blocks of data from the data path from the host device to the storage medium.
9. The storage device of claim 7 , the hardware accelerator to compute message digests comprises the hardware accelerator to separately apply a digest function to the respective blocks of data as the respective blocks of data pass on the data path from the host device to the storage medium.
10. The storage device of claim 7 , comprising the hardware accelerator to also perform compression and security functions for data that passes between the host device and the storage device, the security functions to include encryption, decryption or secure hash computation.
11. The storage device of claim 7 , the hardware accelerator further comprising an offline module, the offline module to:
identify one or more blocks of data previously written to the storage medium;
retrieve the one or more blocks of data and store the one or more blocks of data to a buffer for offline message digest computation;
compute one or more respective message digests of the one or more blocks of data stored to the buffer; and
cause the one or more respective message digests to be sent to the host device to enable the host device to compare the one or more respective message digests with one or more other message digests previously sent from the storage device to detect duplication of blocks of data stored at the storage device.
12. The storage device of claim 7 , the storage device comprises a solid state drive, the storage medium to include NAND memory.
13. A system comprising:
a host device; and
a solid state drive that includes:
an interface coupled with the host device;
a storage medium; and
a hardware accelerator to:
compute message digests for respective blocks of data received from the host device, the hardware accelerator to implement noninvasive snooping of the respective blocks of data to be stored at the storage medium, the noninvasive snooping to occur as the respective blocks of data are received at the storage medium from the host device, the computed message digests are computed by the hardware accelerator via a stateless, flow-through computation applied to the respective blocks of data as the respective blocks of data pass from the host device to the storage medium, the respective blocks of data to traverse a data path from the host device to the storage medium, the hardware accelerator to compute the message digests responsive to separate commands from the host device for respective write requests to store the respective blocks of data in the storage medium, the message digests computed in conjunction with fulfilling respective write requests; and
cause computed message digests to be sent to the host device through the interface,
wherein, the host device to use the computed message digests to detect duplication of blocks of data stored in the storage medium.
14. The system of claim 13 , comprising the hardware accelerator to compute message digests without redirection of the respective blocks of data from the data path from the host device to the storage medium.
15. The system of claim 13 , the hardware accelerator to compute message digests comprises the hardware accelerator to separately apply a digest function to the respective blocks of data as the respective blocks of data pass on the data path from the host device to the storage medium.
16. The system of claim 13 , comprising the hardware accelerator to also perform compression and security functions for data that passes between the host device and the solid state drive, the security functions to include encryption, decryption or secure hash computation.
17. The system of claim 13 , the hardware accelerator further comprising an offline module, the offline module to:
identify one or more blocks of data previously written to the storage medium;
retrieve the one or more blocks of data and store the one or more blocks of data to a buffer for offline message digest computation;
compute one or more respective message digests of the one or more blocks of data stored to the buffer; and
cause the one or more respective message digests to be sent to the host device to enable the host device to compare the one or more respective message digests with one or more other message digests previously sent from the solid state drive to detect duplication of blocks of data stored at the solid state drive.
18. The system of claim 13 , comprising the storage medium to include NAND memory.