Lockless distributed redundant storage and NVRAM caching of compressed data in a highly-distributed shared topology with direct memory access capable interconnect

1. A method for data storage, comprising:

in a system that comprises multiple servers, multiple multi-queue storage devices and at least one storage controller that communicate over a network, storing on the storage devices compressed data belonging to a user volume;

specifying storage locations, in which the compressed data is stored on the storage devices, in a shared data structure that is shared and modified by the servers using remote direct memory access; and

coordinating access to the compressed data by the servers by querying the shared data structure, without executing code on a processor of the storage controller,

wherein storing the compressed data comprises applying a background compression process, which compresses data that has already been stored on the storage devices, and updates the shared data structure so that the servers are able to access the compressed data concurrently and without executing code on the storage controller.

2. The method according to claim 1 , wherein storing the compressed data comprises compressing data by the servers, and sending the compressed data for storage on the storage devices.

3. The method according to claim 1 , wherein storing the compressed data comprises accumulating data blocks, which comprise the compressed data, in a Non-Volatile Random-Access Memory (NVRAM) cache that is accessible to the servers and to the storage devices, so as to form one or more stripes, and transferring the stripes from the NVRAM cache to the storage devices.

4. The method according to claim 3 , wherein storing the compressed data comprises storing multiple compressed blocks of compressed data in one or more data blocks of a stripe, and wherein specifying the storage locations comprises specifying metadata that points to locations of the compressed blocks within the data blocks.

5. The method according to claim 4 , and comprising storing the metadata in the data blocks of the stripe.

6. The method according to claim 4 , and comprising storing the metadata in the shared data structure, separately from the data blocks.

7. The method according to claim 1 , wherein applying the background compression process comprises reading data from one or more data blocks stored on the storage devices, compressing the read data, and rewriting the compressed data back to the storage devices.

8. The method according to claim 7 , wherein rewriting the compressed data comprises compacting the rewritten data by removing regions of invalid data that are present in the data blocks.

9. The method according to claim 1 , wherein storing the compressed data comprises performing compression or decompression in Network Interface Controllers (NICs) of the servers, the storage devices or the storage controller.

10. A system for data storage, comprising:

multiple servers, which are configured to communicate over a network with multiple multi-queue storage devices and with at least one storage controller, to store on the storage devices compressed data belonging to a user volume, to specify storage locations, in which the compressed data is stored on the storage devices, in a shared data structure that is shared and modified by the servers using remote direct memory access, and to coordinate access to the compressed data by the servers by querying the shared data structure, without executing code on a processor of the storage controller,

wherein the servers are further configured to apply a background compression process, which compresses data that has already been stored on the storage devices, and updates the shared data structure so that the servers are able to access the compressed data concurrently and without executing code on the storage controller.

11. The system according to claim 10 , wherein the servers are configured to compress the data and to send the compressed data for storage on the storage devices.

12. The system according to claim 10 , wherein the servers are configured to accumulate data blocks, which comprise the compressed data, in a Non-Volatile Random-Access Memory (NVRAM) cache that is accessible to the servers and to the storage devices, so as to form one or more stripes, and to transfer the stripes from the NVRAM cache to the storage devices.

13. The system according to claim 12 , wherein the servers are configured to store multiple compressed blocks of compressed data in one or more data blocks of a stripe, and to specify the storage locations by specifying metadata that points to locations of the compressed blocks within the data blocks.

14. The system according to claim 13 , wherein the servers are configured to store the metadata in the data blocks of the stripe.

15. The system according to claim 13 , wherein the servers are configured to store the metadata in the shared data structure, separately from the data blocks.

16. The system according to claim 10 , wherein the servers are configured to apply the background compression process by reading data from one or more data blocks stored on the storage devices, compressing the read data, and rewriting the compressed data back to the storage devices.

17. The system according to claim 16 , wherein the servers are configured to compact the rewritten data by removing regions of invalid data that are present in the data blocks.

18. The system according to claim 10 , wherein the servers comprise respective Network Interface Controllers (NICs) that are configured to perform compression or decompression of the data.