Deduplication in a highly-distributed shared topology with direct-memory-access capable interconnect
A method for data storage includes, in a system that includes multiple servers, multiple multi-queue storage devices and at least one storage controller that communicate over a network, storing data blocks by the servers on the storage devices. A shared data structure, which is accessible to the servers using remote direct memory access and which maps hash values calculated over the data blocks to respective storage locations of the data blocks on the storage devices, is maintained. The data blocks stored on the storage device are deduplicated, by looking-up the shared data structure by the servers without executing code on a processor of the storage controller.
1. A method for data storage, comprising:
in a system that comprises multiple servers, multiple multi-queue storage devices and at least one storage controller that communicate over a network, running one or more user applications on a given server among the servers, and storing data blocks, which comprise data produced by the user applications running on the given server, by the given server on the storage devices;
maintaining a shared data structure, which is accessible to the servers using remote direct memory access and which maps hash values calculated over the data blocks to respective storage locations of the data blocks on the storage devices, wherein any hash value found in the shared data structure indicates that a data block that matches the hash value exists in at least one logical volume mapped by at least one of the multiple servers in the system; and
deduplicating, by the given server, the data blocks that comprise the data produced by the user applications running on the given server, by:
looking-up the shared data structure by the given server without executing code on a processor of the storage controller; and
storing in the storage devices only the data blocks that do not exist in any logical volume used by any of the multiple servers in the system, and therefore have no matching hash values in the shared data structure.
2. The method according to claim 1 , wherein the shared data structure comprises a Key-Value (KV) store.
3. The method according to claim 1 , wherein deduplicating the data blocks comprises, when the given server prepares to store a given data block, checking whether the hash value of the given data block already exists in the shared data structure, and if the hash value of the given data block exists, refraining from storing the given data block on the storage devices.
4. The method according to claim 3 , wherein deduplicating the data blocks further comprises, if the hash value of the given data block exists, mapping a logical address of the given data block to a storage location to which the hash value of the given data block is mapped in the shared data structure.
5. The method according to claim 3 , wherein deduplicating the data blocks further comprises, if the hash value of the given data block exists, updating a shared reference count associated with the hash value of the given data block.
6. The method according to claim 3 , wherein deduplicating the data blocks further comprises, if the hash value of the given data block exists, updating a shared bit store to indicate that the given data block is referenced more than once.
7. The method according to claim 1 , wherein deduplicating the data blocks further comprises running a background garbage-collection process that identifies and releases unreferenced data blocks.
8. The method according to claim 1 , further comprising running by the storage controller a background deduplication process that deduplicates data blocks already stored on the storage devices, wherein deduplication operations performed by the background deduplication process are seen atomically by the servers.
9. The method according to claim 1 , wherein deduplicating the data blocks comprises choosing whether or not to deduplicate a given data block, depending on a logical volume to which the given data block belongs.
10. The method according to claim 1 , wherein deduplicating the data blocks comprises calculating the hash values by one or more Network Interface Controllers (NICs) of the servers, the storage devices or the storage controller.
11. A system for data storage, comprising:
at least one storage controller; and
multiple servers, which are configured to communicate over a network with the storage controller and with multiple multi-queue storage devices,
wherein a given server among the servers is configured to:
run one or more user applications;
store data blocks, which comprise data produced by the user applications running on the given server, on the storage devices;
access, using remote direct memory access, a shared data structure that maps hash values calculated over the data blocks to respective storage locations of the data blocks on the storage devices, wherein any hash value found in the shared data structure indicates that a data block that matches the hash value exists in at least one logical volume mapped by at least one of the multiple servers in the system; and
deduplicate the data blocks that comprise the data produced by the user applications running on the given server, by:
looking-up the shared data structure without executing code on a processor of the storage controller; and
storing in the storage devices only the data blocks that do not exist in any logical volume used by any of the multiple servers in the system, and therefore have no matching hash values in the shared data structure.
12. The system according to claim 11 , wherein the shared data structure comprises a Key-Value (KV) store.
13. The system according to claim 11 , wherein the given server is configured to deduplicate the data blocks by, upon preparing to store a given data block, checking whether a hash value of the given data block already exists in the shared data structure, and if the hash value of the given data block exists, refraining from storing the given data block on the storage devices.
14. The system according to claim 13 , wherein the given server is further configured, if the hash value of the given data block exists, to map a logical address of the given data block to a storage location to which the hash value of the given data block is mapped in the shared data structure.
15. The system according to claim 13 , wherein the given server is further configured, if the hash value of the given data block exists, to update a shared reference count associated with the hash value of the given data block.
16. The system according to claim 13 , wherein the given server is further configured, if the hash value of the given data block exists, to update a shared bit store to indicate that the given data block is referenced more than once.
17. The system according to claim 11 , wherein the storage controller is configured to run a background garbage-collection process that identifies and releases unreferenced data blocks.
18. The system according to claim 11 , wherein the storage controller is configured to run a background deduplication process that deduplicates data blocks already stored on the storage devices, wherein deduplication operations performed by the background deduplication process are seen atomically by the servers.
19. The system according to claim 11 , wherein the servers are configured to choose whether or not to deduplicate a given data block, depending on a logical volume to which the given data block belongs.
20. The system according to claim 11 , wherein one or more of: the storage controller and one or more of the servers comprise one or more Network Interface Controllers (NICs), which are configured to communicate over the network and also to calculate the hash values.
21. A server, comprising:
a Network Interface Controller (NIC) configured to connect to a network; and
a processor, which is configured to:
run one or more user applications;
communicate over the network with at least one storage controller and with multiple multi-queue storage devices;
store data blocks, which comprise data produced by the user applications running on the server, on the storage devices;
access, using remote direct memory access, a shared data structure that maps hash values calculated over the data blocks to respective storage locations of the data blocks on the storage devices, wherein any hash value found in the shared data structure indicates that a data block that matches the hash value exists in at least one logical volume mapped by at least one of the multiple servers in the system; and
deduplicate the data blocks that comprise the data produced by the user applications running on the server, by:
looking-up the shared data structure without executing code on any processor of the storage controller; and
storing in the storage devices only the data blocks that do not exist in any logical volume used by any of the multiple servers in the system, and therefore have no matching hash values in the shared data structure.
22. The server according to claim 21 , wherein the processor is configured, upon preparing to store a given data block, to check whether a hash value of the given data block already exists in the shared data structure, and if the hash value exists, to refrain from storing the given data block on the storage devices.