Lockless distributed redundant storage and NVRAM caching of compressed data in a highly-distributed shared topology with direct memory access capable interconnect
A system for data storage includes multiple servers, which are configured to communicate over a network with multiple multi-queue storage devices and with at least one storage controller, to store on the storage devices compressed data belonging to a user volume, to specify storage locations, in which the compressed data is stored on the storage devices, in a shared data structure that is shared and modified by the servers using remote direct memory access, and to coordinate access to the compressed data by the servers by querying the shared data structure, without executing code on a processor of the storage controller.
1. A method for data storage, comprising:
in a system that comprises multiple physical servers, multiple multi-queue storage devices and at least one physical storage controller that communicate over a network, compressing by the servers data that is produced by one or more user applications and belongs to a user volume, sending the compressed data from the servers to the storage devices, and storing the compressed data on the storage devices;
specifying storage locations, in which the compressed data is stored on the storage devices, in a shared data structure that is shared and modified by the servers using remote direct memory access; and
coordinating access to the compressed data by the servers by querying the shared data structure, without executing code on a processor of the storage controller.
2. The method according to claim 1 , wherein storing the compressed data comprises accumulating data blocks, which comprise the compressed data, in a Non-Volatile Random-Access Memory (NVRAM) cache that is accessible to the servers and to the storage devices, so as to form one or more stripes, and transferring the stripes from the NVRAM cache to the storage devices.
3. The method according to claim 2 , wherein storing the compressed data comprises storing multiple compressed blocks of compressed data in one or more data blocks of a stripe, and wherein specifying the storage locations comprises specifying metadata that points to locations of the compressed blocks within the data blocks.
4. The method according to claim 3 , and comprising storing the metadata in the data blocks of the stripe.
5. The method according to claim 3 , and comprising storing the metadata in the shared data structure, separately from the data blocks.
6. The method according to claim 1 , wherein storing the compressed data comprises performing compression or decompression in Network Interface Controllers (NICs) of the servers, the storage devices or the storage controller.
7. The method according to claim 1 ,
wherein specifying the storage locations comprises specifying in the shared data structure a mapping between logical addresses and physical addresses on the storage devices,
wherein querying the shared data structure comprises translating one or more of the logical addresses into the corresponding physical addresses, in the servers by accessing the shared data structure, and
wherein sending the compressed data comprises sending the compressed data, from the servers to the storage devices, in storage commands that specify the physical addresses.
8. A system for data storage, comprising:
multiple physical servers, which are configured to communicate over a network with multiple multi-queue storage devices and with at least one physical storage controller, to compress data that is produced by one or more user applications and belongs to a user volume, to send the compressed data for storage on the storage devices, to specify storage locations, in which the compressed data is stored on the storage devices, in a shared data structure that is shared and modified by the servers using remote direct memory access, and to coordinate access to the compressed data by the servers by querying the shared data structure, without executing code on a processor of the storage controller.
9. The system according to claim 8 , wherein the servers are configured to accumulate data blocks, which comprise the compressed data, in a Non-Volatile Random-Access Memory (NVRAM) cache that is accessible to the servers and to the storage devices, so as to form one or more stripes, and to transfer the stripes from the NVRAM cache to the storage devices.
10. The system according to claim 9 , wherein the servers are configured to store multiple compressed blocks of compressed data in one or more data blocks of a stripe, and to specify the storage locations by specifying metadata that points to locations of the compressed blocks within the data blocks.
11. The system according to claim 10 , wherein the servers are configured to store the metadata in the data blocks of the stripe.
12. The system according to claim 10 , wherein the servers are configured to store the metadata in the shared data structure, separately from the data blocks.
13. The system according to claim 8 , wherein the servers comprise respective Network Interface Controllers (NICs) that are configured to perform compression or decompression of the data.
14. The system according to claim 8 , wherein the servers are configured to:
specify the storage locations by specifying in the shared data structure a mapping between logical addresses and physical addresses on the storage devices;
translate one or more of the logical addresses into the corresponding physical addresses, by accessing the shared data structure; and
send the compressed data to the storage devices in storage commands that specify the physical addresses.