System and method for data deduplication in a smart data accelerator interface device
A hardware accelerator device detects duplicate data blocks independently from a processor of an information handling system. The device includes an interface, a hash function, a bloom filter, and a data comparator. The interface receives data blocks. The hash function determines a hash of a received data block. The bloom filter includes comparison hashes, each associated with a comparison data block. The bloom filter determines if the hash matches a comparison hash. The data comparator directs the interface to receive the comparison block when the hash matches the comparison hash, and compares the received data block with the comparison data block to determine if they are identical. The device stores a pointer when the data comparison block determines that the blocks are identical.
1. A hardware accelerator device to detect duplicate data blocks independently from a processor of an information handling system, the hardware accelerator device comprising:
an interface configured to receive a data block and at least one of a plurality of comparison data blocks;
a hash function accelerator configured to determine a first hash of the received data block;
a bloom filter accelerator including a plurality of comparison hashes, wherein each comparison data block is associated with at least one comparison hash, and wherein the bloom filter accelerator is configured to determine whether the first hash matches a first comparison hash, wherein the first comparison hash is associated with a first comparison data block;
a data comparison accelerator configured, in response to the determination that the first hash matches the first comparison hash, to receive the first comparison data block via the interface, and to compare the received data block with the first comparison data block to determine whether the received data block and the first comparison data block are identical; and
a cache memory block, wherein, in receiving the first comparison data block, the interface is further configured to store the first comparison data blocks to the cache memory device;
wherein, when the data comparison block determines that the received data block and the first comparison data block are identical, the hardware accelerator device is configured to store a first pointer that associates the received data block with the first comparison data block, without storing the received data block.
2. The hardware accelerator device of claim 1 , wherein, when the data comparison block determines that the received data block and the first comparison data block are not identical, the hardware accelerator device is configured to store the first hash in the bloom filter accelerator, and to direct the interface to store the received data block.
3. The hardware accelerator device of claim 1 , wherein:
the bloom filter accelerator is further configured to determine whether or not the first hash matches a second comparison hash, wherein the second comparison hash is associated with a second comparison data block;
the data comparison block is further configured, in response to the determination that the first hash matches the second comparison hash, to receive the second comparison data block via the interface, and to compare the received data block with the second comparison data block to determine whether or not the received data block and the second comparison data block are identical; and
when the data comparison block determines that the received data block and the second comparison data block are identical, the hardware accelerator device is configured to store a second pointer that associates the received data block with the second comparison data block, without storing the received data block.
4. The hardware accelerator device of claim 1 , wherein:
the bloom filter accelerator is further configured to determine that the first hash does not match any of the comparison hashes; and
wherein, when the first hash does not match any of the comparison hashes, the hardware accelerator device is configured to store the first hash in the bloom filter accelerator, and to direct the interface to store the received data block.
5. The hardware accelerator device of claim 1 , wherein:
the hash function accelerator is further configured to determine a second hash of the received data block;
the bloom filter accelerator is further configured to determine whether or not the second hash matches a second comparison hash, wherein the second comparison hash is associated with a second comparison data block; and
the data comparison block is further configured, in response to the determination that the second hash matches the second comparison hash, to receive the second comparison data block, via the interface and to compare the received data block with the second comparison data block to determine whether or not the received data block and the second comparison data block are identical.
6. The hardware accelerator device of claim 5 , wherein in determining the first hash, the hash function accelerator is configured to utilize a first hash function, and, in determining the second hash, the hash function accelerator is configured to utilize a second hash function different from the first hash function.
7. The hardware accelerator device of claim 5 , wherein, in receiving the second comparison data block, the interface is further configured to store the first and second comparison data block to the cache memory device.
8. The hardware accelerator device of claim 7 , wherein the cache memory device is further configured to evict the first comparison data block when the interface stores the second comparison data block in accordance with a cache algorithm.
9. The hardware accelerator device of claim 1 , wherein the interface is a Smart Data Accelerator Interface (SDXI) interface.
10. A method to detect duplicate data blocks independently from a processor of an information handling system, the method comprising:
receiving, by an interface of a hardware accelerator device, a data block and at least one of a plurality of comparison data blocks;
determining, by a hash function accelerator of the hardware accelerator device, a first hash of the received data block;
determining, by a bloom filter accelerator of the hardware accelerator device, whether the first hash matches a first comparison hash, wherein the first comparison hash is associated with a first comparison data block of a plurality of comparison data blocks, wherein the bloom filter accelerator includes a plurality of comparison hashes, and wherein each comparison data block is associated with at least one comparison hash;
receiving, by a data comparison accelerator of the hardware accelerator device, the first comparison data block via the interface in response to determining that the first hash matches the first comparison hash;
comparing, by the data comparison accelerator, the received data block with the first comparison data block to determine whether the received data block and the first comparison data block are identical;
storing a first pointer that associates the received data block with the first comparison data block, without storing the received data block when the data comparison block determines that the received data block and the first comparison data block are identical; and
storing, by the interface, the first comparison data block to a cache memory device of the hardware accelerator device.
11. The method of claim 10 , further comprising:
storing the first hash in the bloom filter accelerator when the received data block and the first comparison data block are not identical; and
directing the interface to store the received data block.
12. The method of claim 10 , further comprising:
determining whether or not the first hash matches a second comparison hash, wherein the second comparison hash is associated with a second comparison data block;
receiving the second comparison data block in response to determining that the first hash matches the second comparison hash;
comparing the received data block with the second comparison data block to determine whether or not the received data block and the second comparison data block are identical; and
storing a second pointer that associates the received data block with the second comparison data block without storing the received data block when the received data block and the second comparison data block are identical.
13. The method of claim 10 , further comprising:
determining that the first hash does not match any of the comparison hashes;
storing the first hash in the bloom filter accelerator when the first hash does not match any of the comparison hashes; and
directing the interface to store the received data block.
14. The method of claim 10 , further comprising:
determining a second hash of the received data block;
determining whether or not the second hash matches a second comparison hash, wherein the second comparison hash is associated with a second comparison data block;
receiving receive the second comparison data block in response to determining that the second hash matches the second comparison hash; and
comparing the received data block with the second comparison data block to determine whether or not the received data block and the second comparison data block are identical.
15. The method of claim 14 , wherein in determining the first hash, the hash function accelerator is configured to utilize a first hash function, and, in determining the second hash, the hash function accelerator is configured to utilize a second hash function different from the first hash function.
16. The method of claim 14 , further comprising:
storing, by the interface, the second comparison data block to the cache memory device.
17. The method of claim 16 , further comprising:
evicting, by the cache memory device, the first comparison data block when the interface stores the second comparison block in accordance with a cache algorithm.
18. The method of claim 10 , wherein the interface is a Smart Data Accelerator Interface (SDXI) interface.
19. An information handling system, comprising:
a processor; and
a hardware accelerator device to detect duplicate data blocks independently from the processor, the hardware accelerator device including:
an interface configured to receive a data block and at least one of a plurality of comparison data blocks;
a hash function accelerator configured to determine a first hash of the received data block;
a bloom filter accelerator including a plurality of comparison hashes, wherein each comparison data block is associated with at least one comparison hash, and wherein the bloom filter accelerator is configured to determine whether the first hash matches a first comparison hash, wherein the first comparison hash is associated with a first comparison data block;
a data comparison accelerator configured, in response to the determination that the first hash matches the first comparison hash, to receive the first comparison data block via the interface, and to compare the received data block with the first comparison data block to determine whether the received data block and the first comparison data block are identical; and
a cache memory block, wherein, in receiving the first comparison data block, the interface is further configured to store the first comparison data blocks to the cache memory device;
wherein, when the data comparison block determines that the received data block and the first comparison data block are identical, the hardware accelerator device is configured to store a first pointer that associates the received data block with the first comparison data block, without storing the received data block.
20. The information handling system of claim 19 , wherein, when the data comparison block determines that the received data block and the first comparison data block are not identical, the hardware accelerator device is configured to store the first hash in the bloom filter accelerator, and to direct the interface to store the received data block.