Method and systems for storing data in a storage pool using memory semantics with applications interacting with emulated block devices
A method for storing data, comprising receiving, by a file system (FS) client executing on an offload component, a first request from a translation module, wherein the translation module translated a second request that was to be performed on an emulated block device into the first request, wherein the first request is specified using file semantics, wherein the first request is associated with data, wherein the offload component is located in a hardware layer of a client application node, and wherein the translation module is located on the offload component, and processing the first request by the FS client and a memory hypervisor module, wherein the FS client and the memory hypervisor module are executing in a modified client FS container on the offload component, wherein processing the first request results in at least a portion of the data being stored in a location in a storage pool.
1. A method for storing data, the method comprising:
monitoring, by a translation module, emulated block devices for requests issued thereto;
issuing, by a non-volatile memory express (NVMe) controller, a request to an emulated block device of the emulated block devices, the request being associated with data;
translating, by the translation module and in response to the monitoring, the request into a second request specified using file semantics,
receiving, by a file system (FS) client, the second request from the translation module; and
processing, by the FS client and a memory hypervisor module, the second request to store at least a portion of the data in a location in a storage pool,
wherein the NVMe controller, the emulated block devices, the translation module, FS client, and the memory hypervisor module all operate in a modified client FS container executing on an offload component,
wherein the offload component resides in a hardware layer of a client application node.
2. The method of claim 1 , wherein the offload component emulates the NVMe controller and the emulated block devices.
3. The method of claim 1 , wherein the request was initiated by an application executing in an application container on the client application node.
4. The method of claim 1 , wherein processing the second request comprises:
generating, by the memory hypervisor module, at least one input/output (I/O) request associated with the second request,
wherein the at least one I/O request specifies the location in the storage pool,
wherein the location is determined using a data layout; and
issuing, by the memory hypervisor module, the at least one I/O request to the storage pool.
5. The method of claim 4 , wherein the data layout is obtained from a metadata node.
6. The method of claim 1 , wherein the location in the storage pool is a location in a memory device.
7. The method of claim 6 , wherein the memory device is located on a storage node.
8. The method of claim 7 , wherein the storage pool comprises the memory device, a second memory device on a metadata node, and a third memory device on the client application node.
9. The method of claim 6 , wherein the memory device is a persistent memory (PMEM) device.
10. A non-transitory computer readable medium comprising instructions which, when executed by a processor, enables the processor to perform a method for storing data, the method comprising:
monitoring, by a translation module, emulated block devices for requests issued thereto;
issuing, by a non-volatile memory express (NVMe) controller, a request to an emulated block device of the emulated block devices, the request being associated with data;
translating, by the translation module and in response to the monitoring, the request into a second request specified using file semantics,
receiving, by a file system (FS) client, the second request from the translation module; and
processing, by the FS client and a memory hypervisor module, the second request to store at least a portion of the data in a location in a storage pool,
wherein the NVMe controller, the emulated block devices, the translation module, FS client, and the memory hypervisor module all operate in a modified client FS container executing on an offload component,
wherein the offload component resides in a hardware layer of a client application node.
11. The non-transitory computer readable medium of claim 10 , wherein the offload component emulates the NVMe controller and the emulated block devices.
12. The non-transitory computer readable medium of claim 10 , wherein the request was initiated by an application executing in an application container on the client application node.
13. The non-transitory computer readable medium of claim 10 , wherein processing the second request comprises:
generating, by the memory hypervisor module, at least one input/output (I/O) request associated with the second request,
wherein the at least one I/O request specifies the location in the storage pool,
wherein the location is determined using a data layout; and
issuing, by the memory hypervisor module, the at least one I/O request to the storage pool.
14. The non-transitory computer readable medium of claim 13 , wherein the data layout is obtained from a metadata node.
15. The non-transitory computer readable medium of claim 10 , wherein the location in the storage pool is a location in a memory device.
16. The non-transitory computer readable medium of claim 15 , wherein the storage pool comprises the memory device, a second memory device on a metadata node, and a third memory device on the client application node.
17. The non-transitory computer readable medium of claim 15 , wherein the memory device is located on a storage node and wherein the memory device is a persistent memory (PMEM) device.
18. An offload component in a hardware layer of a client application node, comprising:
a physical processing component;
a modified client file system (FS) container executing on the physical processing component; and
a non-volatile memory express (NVMe) controller, emulated block devices, a translation module, a FS client, and a memory hypervisor module all operating in the modified client FS container,
wherein the NVMe controller is configured to:
issue a request to an emulated block device of the emulated block devices, the request being associated with data,
wherein the translation module is configured to:
monitor the emulated block devices for requests issued thereto; and
translate, in response to monitoring the emulated block devices, the request into a second request specified using file semantics,
wherein the FS client is configured to:
receive the second request from the translation module,
wherein the memory hypervisor module is configured to:
generate at least one input/output (I/O) request associated with the second request,
wherein the at least one I/O request specifies the location in a storage pool,
wherein the location is determined using a data layout obtained from a metadata node; and
issue the at least one I/O request to the storage pool,
wherein processing the at least one I/O request results in at least a portion of the data being stored in a location in the storage pool.