Methods and systems for storing data in a distributed system using offload components
A method for storing data, the method comprising receiving, by an offload component in a client application node, a request originating from an application executing in an application container on the client application node, wherein the request is associated with data and wherein the offload component is located in a hardware layer of the client application node, and processing, by the offload component, the request by a file system (FS) client and a memory hypervisor module executing in a modified client FS container on the offload component, wherein processing the request results in at least a portion of the data in a location in a storage pool.
1. A method for storing data, the method comprising:
receiving, by an offload component in a client application node, a request originating from an application executing in an application container on the client application node, wherein the request is associated with data and wherein the offload component is located in a hardware layer of the client application node; and
processing, by the offload component, the request by a file system (FS) client and a memory hypervisor module executing in a modified client FS container on the offload component, wherein processing the request results in at least a portion of the data processed by the offload component being stored in a location in a storage pool,
wherein the request is received by a bi-directional bridging component in the modified client FS container,
wherein the request was issued by a second bi-directional bridging component executing in a client bridging container,
wherein the application container and the client bridging container are executing in a user space of the client application node, and
wherein the request from the application is received by an operating system executing on the client application node and redirected to the client bridging container.
2. The method of claim 1 , wherein processing the request comprises:
generating, by the memory hypervisor module, at least one input/output (I/O) request associated with the request, wherein the at least one I/O request specifies the location in the storage pool, wherein the location is determined using a data layout; and
issuing, by the memory hypervisor module, the at least one I/O request to the storage pool.
3. The method of claim 2 , wherein the data layout is obtained from a metadata node.
4. The method of claim 1 , wherein the location in the storage pool is a location in a memory device.
5. The method of claim 4 , wherein the memory device is located on a storage node.
6. The method of claim 5 , wherein the storage pool comprises the memory device, a second memory device on a metadata node, and a third memory device on the client application node.
7. The method of claim 4 , wherein the memory device is a persistent memory (PMEM) device.
8. A non-transitory computer readable medium comprising instructions which, when executed by a processor, enables the processor to perform a method for storing data, the method comprising;
receiving, by an offload component in a client application node, a request originating from an application executing in an application container on the client application node, wherein the request is associated with data and wherein the offload component is located in a hardware layer of the client application node; and
processing, by the offload component, the request by a file system (FS) client and a memory hypervisor module executing in a modified client FS container on the offload component, wherein processing the request results in at least a portion of the data processed by the offload component being stored in a location in a storage pool,
wherein the request is received by a bi-directional bridging component in the modified client FS container,
wherein the request was issued by a second bi-directional bridging component executing in a client bridging container,
wherein the application container and the client bridging container are executing in a user space of the client application node, and
wherein the request from the application is received by an operating system executing on the client application node and redirected to the client bridging container.
9. The non-transitory computer readable medium of claim 8 , wherein processing the request comprises:
generating, by the memory hypervisor module, at least one input/output (I/O) request associated with the request, wherein the at least one I/O request specifies the location in the storage pool, wherein the location is determined using a data layout; and
issuing, by the memory hypervisor module, the at least one I/O request to the storage pool.
10. The non-transitory computer readable medium of claim 9 , wherein the data layout is obtained from a metadata node.
11. The non-transitory computer readable medium of claim 8 , wherein the location in the storage pool is a location in a memory device.
12. The non-transitory computer readable medium of claim 11 , wherein the memory device is located on a storage node and wherein the memory device is a persistent memory (PMEM) device.
13. The method of claim 5 , wherein the storage pool comprises the memory device, a second memory device on a metadata node, and a third memory device on the client application node.
14. A node, comprising:
a hardware processor;
application container executing on the hardware processor;
an offload component comprising a physical processing component, wherein the offload component is configured to:
receive a request originating from an application, wherein the request is associated with data;
process, by the physical processing component, the request by a file system (FS) client and a memory hypervisor module executing in a modified client FS container on the offload component, wherein processing the request results in at least a portion of the data processed by the offload component being stored in a location in a storage pool,
wherein the request is received by a bi-directional bridging component in the modified client FS container,
wherein the request was issued by a second bi-directional bridging component executing in a client bridging container,
wherein the application container and the client bridging container are executing in a user space of the client application node, and
wherein the request from the application is received by an operating system executing on the client application node and redirected to the client bridging container.