Methods and systems for seamless tiering in a distributed storage system
A method for processing requests, the method comprising: receiving, from a client application node and by a metadata node, a request for a data layout, determining a QoS tag associated with the request, identifying at least one file system block (FSB) in a sparse virtual space, wherein the FSB is mapped to a location in a storage pool that is associated with the QoS tag, generating the data layout based on the at least one identified FSB; and providing the data layout to the client application, wherein the client application node uses the data layout to directly write data to the location in the storage pool, wherein the metadata node does not participate in the writing of the data to the location in the storage pool.
1. A method for processing requests, the method comprising:
receiving, from a client application node (CAN) and by a metadata node, a request for a data layout;
determining a QoS tag associated with the request;
identifying at least one file system block (FSB) in a sparse virtual space, wherein the FSB is mapped to a location in a storage pool that is associated with the QoS tag;
generating the data layout based on the at least one identified FSB; and
providing the data layout to the CAN, wherein the CAN uses the data layout to directly write data to the location in the storage pool,
wherein a redundant array of independent disks is implemented in the storage pool to protect the data,
wherein the metadata node does not participate in the writing of the data to the location in the storage pool to allow the request to be performed with low latency and without impacting a writing performance of the data, wherein the CAN calculates a fingerprint for the data,
wherein the CAN encrypts the data using an encryption key scheme to limit which encryption key is used to encrypt a first portion of the data and a second portion of the data, wherein a per-snapshot static encryption key (PSSEK) is used to encrypt the first portion and the second portion of the data, wherein the PSSEK is generated based on a state snapshot of the sparse virtual space,
wherein the CAN calculates, using a cyclic redundancy check (CRC), a (CRC) value for the data, wherein the CAN provides the fingerprint and the CRC value to the metadata node, wherein the metadata node commits the FSB by storing the CRC value, wherein, based on a presence of the fingerprint in a self-balancing tree data structure, the metadata node determines that the data is deduplicable,
wherein, based on determining the data is deduplicable, the metadata node updates a mapping between a [file, offset] and a [scale out volume, offset] to reflect that the data is deduplicated and to make sure that logical blocks that are associated with the data are mapped to the same FSB,
wherein the logical blocks are specified in the data layout,
wherein, using the data layout, the CAN translates the [file, offset] into the [scale out volume, offset],
wherein, based on the updating, the metadata node marks the data for a garbage collection.
2. The method of claim 1 , wherein determining the QoS tag comprises analyzing a logical hierarchy of a file system to determine the QoS tag associated with a [file, offset] specified in the request for the data layout, wherein the file system manages the sparse virtual space.
3. The method of claim 2 , wherein the logical hierarchy of the file system comprises a plurality of elements, wherein an element of the plurality of elements is associated with the QoS tag, and wherein the element corresponds to the file identified in the [file, offset].
4. The method of claim 2 , wherein the logical hierarchy of the file system comprises a plurality of elements, wherein an element of the plurality of elements is associated with the QoS tag, wherein the element is indirectly connected to a second element of the plurality of elements, and wherein the second element corresponds to the file identified in the [file, offset].
5. The method of claim 1 , wherein the data layout provides the mapping between the [file, offset] and the [scale out volume, offset], wherein the request for the data layout specifies the [file, offset], wherein the location in the storage pool is mapped to the [scale out volume, offset], and wherein the FSB is mapped to the [file, offset] and the [scale out volume, offset].
6. The method of claim 1 , wherein the sparse virtual space is data structure that is initially allocated substantially sparse to handle future allocations of file system blocks without having to modify a size of the sparse virtual space.
7. The method of claim 1 , wherein the location in the storage pool is a location in a memory device.
8. The method of claim 7 , wherein the memory device is located on a storage node.
9. The method of claim 8 , wherein the storage pool comprises the memory device, a second memory device on the metadata node, and a third memory device on the CAN.
10. The method of claim 7 , wherein the memory device is a persistent memory (PMEM) device.
11. A non-transitory computer readable medium comprising instructions which, when executed by a processor, enables the processor to perform a method for processing requests, the method comprising:
receiving, from a client application node (CAN) and by a metadata node, a request for a data layout;
determining a QoS tag associated with the request;
identifying at least one file system block (FSB) in a sparse virtual space, wherein the FSB is mapped to a location in a storage pool that is associated with the QoS tag;
generating the data layout based on the at least one identified FSB; and
providing the data layout to the CAN, wherein the CAN uses the data layout to directly write data to the location in the storage pool,
wherein a redundant array of independent disks is implemented in the storage pool to protect the data,
wherein the metadata node does not participate in the writing of the data to the location in the storage pool to allow the request to be performed with low latency and without impacting a writing performance of the data, wherein the CAN calculates a fingerprint for the data,
wherein the CAN encrypts the data using an encryption key scheme to limit which encryption key is used to encrypt a first portion of the data and a second portion of the data, wherein a per-snapshot static encryption key (PSSEK) is used to encrypt the first portion and the second portion of the data, wherein the PSSEK is generated based on a state snapshot of the sparse virtual space,
wherein the CAN calculates, using a cyclic redundancy check (CRC), a (CRC) value for the data, wherein the CAN provides the fingerprint and the CRC value to the metadata node, wherein the metadata node commits the FSB by storing the CRC value, wherein, based on a presence of the fingerprint in a self-balancing tree data structure, the metadata node determines that the data is deduplicable,
wherein, based on determining the data is deduplicable, the metadata node updates a mapping between a [file, offset] and a [scale out volume, offset] to reflect that the data is deduplicated and to make sure that logical blocks that are associated with the data are mapped to the same FSB,
wherein the logical blocks are specified in the data layout,
wherein, using the data layout, the CAN translates the [file, offset] into the [scale out volume, offset],
wherein, based on the updating, the metadata node marks the data for a garbage collection.
12. The non-transitory computer readable medium of claim 11 , wherein determining the QoS tag comprises analyzing a logical hierarchy of a file system to determine the QoS tag associated with a [file, offset] specified in the request for the data layout, wherein the file system manages the sparse virtual space.
13. The non-transitory computer readable medium of claim 12 , wherein the logical hierarchy of the file system comprises a plurality of elements, wherein an element of the plurality of elements is associated with the QoS tag, and wherein the element corresponds to the file identified in the [file, offset].
14. The non-transitory computer readable medium of claim 12 , wherein the logical hierarchy of the file system comprises a plurality of elements, wherein an element of the plurality of elements is associated with the QoS tag, wherein the element is indirectly connected to a second element of the plurality of elements, and wherein the second element corresponds to the file identified in the [file, offset].
15. The non-transitory computer readable medium of claim 11 , wherein the data layout provides the mapping between the [file, offset] and the [scale out volume, offset], wherein the request for the data layout specifies the [file, offset], wherein the location in the storage pool is mapped to the [scale out volume, offset], and wherein the FSB is mapped to the [file, offset] and the [scale out volume, offset].
16. The non-transitory computer readable medium of claim 11 , wherein the sparse virtual space is data structure that is initially allocated substantially sparse to handle future allocations of file system blocks without having to modify a size of the sparse virtual space.
17. The non-transitory computer readable medium of claim 11 , wherein the location in the storage pool is a location in a memory device.
18. The non-transitory computer readable medium of claim 17 , wherein the memory device is located on a storage node and wherein the memory device is a persistent memory (PMEM) device.
19. The non-transitory computer readable medium of claim 18 , wherein the storage pool comprises the memory device, a second memory device on the metadata node, and a third memory device on the CAN.
20. A metadata node, comprising:
memory,
a processor, wherein the processor is configured to:
receive, from a client application node (CAN), a request for a data layout;
determine a QoS tag associated with the request;
identify at least one file system block (FSB) in a sparse virtual space, wherein the FSB is mapped to a location in a storage pool that is associated with the QoS tag;
generate the data layout based on the at least one identified FSB; and
provide the data layout to the CAN, wherein the CAN uses the data layout to directly write data to the location in the storage pool,
wherein a redundant array of independent disks is implemented in the storage pool to protect the data,
wherein the metadata node does not participate in the writing of the data to the location in the storage pool to allow the request to be performed with low latency and without impacting a writing performance of the data, wherein the CAN calculates a fingerprint for the data,
wherein the CAN encrypts the data using an encryption key scheme to limit which encryption key is used to encrypt a first portion of the data and a second portion of the data, wherein a per-snapshot static encryption key (PSSEK) is used to encrypt the first portion and the second portion of the data, wherein the PSSEK is generated based on a state snapshot of the sparse virtual space,
wherein the CAN calculates, using a cyclic redundancy check (CRC), a (CRC) value for the data, wherein the CAN provides the fingerprint and the CRC value to the metadata node, wherein the metadata node commits the FSB by storing the CRC value,
wherein, based on a presence of the fingerprint in a self-balancing tree data structure, the metadata node determines that the data is deduplicable, wherein, based on determining the data is deduplicable, the metadata node updates a mapping between a [file, offset] and a [scale out volume, offset] to reflect that the data is deduplicated and to make sure that logical blocks that are associated with the data are mapped to the same FSB,
wherein the logical blocks are specified in the data layout,
wherein, using the data layout, the CAN translates the [file, offset] into the [scale out volume, offset],
wherein, based on the updating, the metadata node marks the data for a garbage collection.