Methods and systems for methods and systems for in-line deduplication in a distributed storage system
A method for deduplicating data comprising: obtaining, from a metadata node and by file system client executing on a client application node, a data layout; generating, by the client application node, a fingerprint for the data stored on the client application node; generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request specifying a location in a storage pool, wherein the location is determined using the data layout; issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the data being stored at the location; and after issuing the at least one I/O request to the storage pool, transmitting the fingerprint to the metadata node, wherein the metadata node attempts to deduplicate the data using the fingerprint.
1. A method for storing data, the method comprising:
obtaining, from a metadata node and by file system (FS) client executing on a client application node, a data layout;
generating, by the client application node, a fingerprint for the data stored on the client application node;
generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request, wherein the at least one I/O request specifies a location in a storage pool, wherein the location is determined using the data layout;
issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the data being stored at the location; and
after issuing the at least one I/O request to the storage pool, transmitting the fingerprint to the metadata node, wherein the metadata node attempts to deduplicate the data using the fingerprint.
2. The method of claim 1 , further comprising:
receiving a request to write the data to the storage pool, wherein the data is associated with a [file, offset], and wherein the data layout provides a mapping between the a [file, offset] and a [scale out volume, offset].
3. The method of claim 1 , further comprising:
issuing, by the client application node, a request to commit metadata associated with the data to the metadata node, wherein the metadata comprises the data layout and the fingerprint.
4. The method of claim 1 , wherein the location in the storage pool is a location in a memory device.
5. The method of claim 4 , wherein the memory device is located on a storage node.
6. The method of claim 5 , wherein the storage pool comprises the memory device, a second memory device on the metadata node, and a third memory device on the client application node.
7. The method of claim 5 , wherein the at least one I/O request is issued directly to the storage node and is not processed by the metadata node.
8. The method of claim 4 , wherein the memory device is a persistent memory (PMEM) device.
9. A non-transitory computer readable medium comprising instructions which, when executed by a processor, enables the processor to perform a method for storing data, the method comprising:
obtaining, from a metadata node and by file system (FS) client executing on a client application node, a data layout;
generating, by the client application node, a fingerprint for the data stored on the client application node;
generating, by a memory hypervisor module executing on the client application node, at least one input/output (I/O) request, wherein the I/O request specifies a location in a storage pool, wherein the location is determined using the data layout;
issuing, by the memory hypervisor module, the at least one I/O request to the storage pool, wherein processing the at least one I/O request results in at least a portion of the encrypted data being stored at the location; and
after issuing the at least one I/O request to the storage pool, transmitting the fingerprint to the metadata node, wherein the metadata node attempts to deduplicate the data using the fingerprint.
10. The non-transitory computer readable medium of claim 9 , the method further comprising:
receiving a request to write the data to the storage pool, wherein the data is associated with a [file, offset], and wherein the data layout provides a mapping between the a [file, offset] and a [scale out volume, offset].
11. The non-transitory computer readable medium of claim 9 , the method further comprising:
issuing, by the client application node, a request to commit metadata associated with the data to the metadata node, wherein the metadata comprises the data layout and the fingerprint.
12. The non-transitory computer readable medium of claim 9 , wherein the location in the storage pool is a location in a memory device.
13. The non-transitory computer readable medium of claim 12 , wherein the memory device is located on a storage node.
14. The non-transitory computer readable medium of claim 13 , wherein the storage pool comprises the memory device, a second memory device on the metadata node, and a third memory device on the client application node.
15. The non-transitory computer readable medium of claim 13 , wherein the at least one I/O request is issued directly to the storage node and is not processed by the metadata node.
16. The non-transitory computer readable medium of claim 12 , wherein the memory device is a persistent memory (PMEM) device.
17. A system, comprising:
a client application node comprising a processor and configured to:
obtain a data layout from a metadata node;
generate a fingerprint for the data stored on the client application node;
generate at least one input/output (I/O) request, wherein the I/O request specifies a location in a storage pool, wherein the location is determined using the data layout, wherein the storage pool comprises a memory device in a storage node and wherein the location is in the memory device;
issue the at least one I/O request to the storage node; and
after issuing the at least one I/O request to the storage pool, transmit the fingerprint to the metadata node;
the metadata node comprising a second processor and configured to:
provide the data layout to the client application node; and
attempt to deduplicate the data stored in the storage pool using the fingerprint; and
the storage node comprising a third processor, the memory device and configured to:
process the at least one I/O request, wherein processing the at least one I/O request results in at least a portion of the data being stored at the location.
18. The system of claim 17 , wherein the storage pool further comprises a second memory device on the metadata node and a third memory device on the client application node.
19. The system of claim 17 , wherein the at least one I/O request is issued directly to the storage node and is not processed by the metadata node.
20. The system of claim 17 , wherein the memory device is a persistent memory (PMEM) device.