Methods and systems for error detection and correction in a distributed storage system
A method for storing data, comprising: obtaining, from a metadata node and by file system client executing on a client application node, a data layout, generating, by the client application node, an error detection value (EDV) for the data stored on the client application node; generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request specifying a location in a storage pool, wherein the location is determined using the data layout; issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the data being stored at the location; and after issuing the at least one I/O request to the storage pool, transmitting the EDV to the metadata node, wherein the metadata node stores the EDV.
1. A method for storing data, the method comprising:
obtaining, from a metadata node and by file system (FS) client executing on a client application node, a data layout;
generating, by the client application node, an error detection value (EDV) for the data stored on the client application node;
generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request, wherein the at least one I/O request specifies a location in a storage pool, wherein the location is determined using the data layout;
issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the data being stored at the location; and
after issuing the at least one I/O request to the storage pool, transmitting the EDV to the metadata node, wherein the metadata node stores the EDV, wherein the metadata node subsequently provides the EDV to at least one client application node in response to the at least one client application node requesting the data layout, and wherein the EDV is used by the at least one client application node to detect whether there are any errors in the data.
2. The method of claim 1 , further comprising:
receiving a request to write the data to the storage pool, wherein the data is associated with a [file, offset], and wherein the data layout provides a mapping between the a [file, offset] and a [scale out volume, offset].
3. The method of claim 1 , further comprising:
issuing, by the client application node, a request to commit metadata associated with the data to the metadata node, wherein the metadata comprises the data layout, and wherein the EDV is transmitted with the request to commit the metadata.
4. The method of claim 1 , wherein the location in the storage pool is a location in a memory device.
5. The method of claim 4 , wherein the memory device is located on a storage node.
6. The method of claim 5 , wherein the storage pool comprises the memory device, a second memory device on the metadata node, and a third memory device on the client application node.
7. The method of claim 5 , wherein the at least one I/O request is issued directly to the storage node and is not processed by the metadata node.
8. The method of claim 4 , wherein the memory device is a persistent memory (PMEM) device.
9. A non-transitory computer readable medium comprising instructions which, when executed by a processor, enables the processor to perform a method for storing data, the method comprising:
obtaining, from a metadata node and by file system (FS) client executing on a client application node, a data layout;
generating, by the client application node, an error detection value (EDV) for the data stored on the client application node;
generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request, wherein the I/O request specifies a location in a storage pool, wherein the location is determined using the data layout;
issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the encrypted data being stored at the location; and
after issuing the at least one I/O request to the storage pool, transmitting the EDV to the metadata node, wherein the metadata node stores the EDV, wherein the metadata node subsequently provides the EDV to at least one client application node in response to the at least one client application node requesting the data layout, and wherein the EDV is used by the at least one client application node to detect whether there are any errors in the data.
10. The non-transitory computer readable medium of claim 9 , the method further comprising:
receiving a request to write the data to the storage pool, wherein the data is associated with a [file, offset], and wherein the data layout provides a mapping between the a [file, offset] and a [scale out volume, offset].
11. The non-transitory computer readable medium of claim 9 , the method further comprising:
issuing, by the client application node, a request to commit metadata associated with the data to the metadata node, wherein the metadata comprises the data layout, and wherein the EDV is transmitted with the request to commit the metadata.
12. The non-transitory computer readable medium of claim 9 , wherein the location in the storage pool is a location in a memory device.
13. The non-transitory computer readable medium of claim 12 , wherein the memory device is located on a storage node.
14. The non-transitory computer readable medium of claim 13 , wherein the storage pool comprises the memory device, a second memory device on the metadata node, and a third memory device on the client application node.
15. The non-transitory computer readable medium of claim 13 , wherein the at least one I/O request is issued directly to the storage node and is not processed by the metadata node.
16. The non-transitory computer readable medium of claim 12 , wherein the memory device is a persistent memory (PMEM) device.
17. A system, comprising:
a client application node comprising a processor and configured to:
obtain a data layout from a metadata node;
generate an error detection value (EDV) for the data stored on the client application node;
generate at least one input/output (I/O) request, wherein the I/O request specifies a location in a storage pool, wherein the location is determined using the data layout, wherein the storage pool comprises a memory device in a storage node and wherein the location is in the memory device;
issue the at least one I/O request to the storage node; and
after issuing the at least one I/O request to the storage pool, transmit the EDV to the metadata node;
the metadata node comprising a second processor and configured to:
provide the data layout to the client application node;
store the EDV;
provide the EDV to at least one client application node in response to the at least one client application node requesting the data layout, wherein the EDV is used by the at least one client application node to detect whether there are any errors in the data; and
the storage node comprising a third processor, the memory device and configured to:
process the at least one I/O request, wherein processing the at least one I/O request results in at least a portion of the data being stored at the location.
18. The system of claim 17 , wherein the storage pool further comprises a second memory device on the metadata node and a third memory device on the client application node.
19. The system of claim 17 , wherein the at least one I/O request is issued directly to the storage node and is not processed by the metadata node.
20. The system of claim 17 , wherein the memory device is a persistent memory (PMEM) device.