Dynamic storage journaling partitions for efficient resource use and improved system throughput
Systems and methods are for improving latency and throughput for metadata-heavy workloads and/or workloads including metadata bursts by decoupling data journal records and metadata-only journal records are provided. According to one embodiment, expedited and independent space reclamation is facilitated by differentiating between various types of journal records chains of which should be retained until different conditions are met. For example, data journal records may be added to data journal record chains within a persistent KV store and metadata-only journal records may be added to metadata-only journal record chains within the persistent KV store. Reclamation of spaced utilized by a data journal record chain may be reclaimed after both remote node data flush has been completed and the completion of a local CP for all records in the chain, whereas records of a metadata-only journal chain may be freed independently upon completion of a local CP for all records.
1 . A method comprising:
maintaining, within a cluster of a plurality of nodes representing a distributed storage system, an operation log of file system operations that have yet to be flushed to storage of the distributed storage system, including (i) decoupling a plurality of types of journal records from each other by organizing the operation log as a plurality of chains of journal records in which all journal records of a given chain of the plurality of chains of journal records are of a same type of the plurality of types of journal records and (ii) storing the operation log within a persistent key-value (KV) store, wherein the plurality of types of journal records include (a) a data journal record having both data and metadata of a given file system operation at issue and (b) a metadata-only journal record having metadata of the given file system operation at issue;
receiving a storage request by a node of the plurality of nodes;
dynamically determining, by the node, based on an operation associated with the storage request, a type of journal record to be used to log the operation within the persistent KV store from among the plurality of types of journal records;
based on the determined type of journal record being the metadata-only journal record, logging, by the node, the operation and corresponding metadata within a new or an active chain of metadata-only journal records within the persistent KV store;
based on the determined type of journal record being the data journal record, logging, by the node, the operation, corresponding data, and the corresponding metadata within a new or an active chain of data journal records within the persistent KV store; and
based on their respective different sets of one or more reclamation dependencies, independently reclaiming space in the persistent KV store associated with a plurality of metadata-only journal records and a plurality of data journal records of respective frozen chains of a plurality of frozen chains within the persistent KV store.
2 . The method of claim 1 , further comprising responsive to receipt of an indication regarding completion of a consistency point (CP) by the node, causing, by the node, the space used by the plurality of metadata-only journal records of a first frozen chain of the plurality of chains associated with a prior CP to be reclaimed by the persistent KV store.
3 . The method of claim 1 , further comprising:
after receipt of an indication that processing of data and metadata flushed to a remote node of the plurality of nodes from a data journal record of the plurality of data journal records of a second frozen chain of the plurality of frozen chains has been completed by the remote node, determining, by the node, whether processing of data and metadata of all of the plurality of data journal records has been completed; and
after an affirmative determination, causing, by the node, the space used by the plurality of data journal records to be reclaimed by the persistent KV store.
4 . The method of claim 1 , wherein the persistent KV store is stored within a non-volatile random access memory (NVRAM) of the cluster.
5 . The method of claim 1 , wherein the plurality of nodes are hosted within a container orchestration platform.
6 . The method of claim 1 , wherein the persistent KV store is stored within a storage device associated with the cluster.
7 . A non-transitory machine readable medium storing instructions, which when executed by one or more processing resources of a node of a plurality of nodes of a cluster representing a distributed storage system, cause the node to:
maintain an operation log of file system operations that have yet to be flushed to storage of the distributed storage system, including (i) decoupling a plurality of types of journal records from each other by organizing the operation log as a plurality of chains of journal records in which all journal records of a given chain of the plurality of chains of journal records are of a same type of the plurality of types of journal records and (ii) storing the operation log within a persistent key-value (KV) store, wherein the plurality of types of journal records include (a) a data journal record having both data and metadata of a given file system operation at issue and (b) a metadata-only journal record having metadata of the given file system operation at issue;
receive a storage request from a client of the distributed storage system or a file system of the node;
determine a type of journal record to be used to log an operation associated with the storage request within the persistent KV store from among the plurality of types of journal records;
based on determining the type of journal record to be used is the metadata-only journal record, log the operation and corresponding metadata within a new or an active chain of metadata-only journal records within the persistent KV store; and
based on determining the type of journal record to be used is the data journal record, log the operation, corresponding data, and the corresponding metadata within a new or an active chain of data journal records within the persistent KV store; and
based on their respective different sets of one or more reclamation dependencies, independently reclaim space in the persistent KV store associated with a plurality of metadata-only journal records and a plurality of data journal records of respective frozen chains of a plurality of frozen chains within the persistent KV store.
8 . The non-transitory machine readable medium of claim 7 , wherein the instructions further cause the node to responsive to completion of a local consistency point (CP), issue a call directing the persistent KV store to reclaim the space used by the plurality of metadata-only journal records of a first frozen chain of the plurality of chains associated with the local CP.
9 . The non-transitory machine readable medium of claim 7 , wherein the instructions further cause the node to:
after completion of flushing of data and metadata to a remote node of the plurality of nodes from a data journal record of the plurality of data journal records of a second frozen chain of the plurality of frozen chains, determine whether processing of data and metadata of all of the plurality of data journal records has been completed; and
after an affirmative determination, issue a call directing the persistent KV store to reclaim the space used by the plurality of data journal records of the second frozen chain.
10 . The non-transitory machine readable medium of claim 7 , wherein the persistent KV store is stored within a non-volatile random access memory (NVRAM) of the cluster.
11 . The non-transitory machine readable medium of claim 7 , wherein the plurality of nodes are hosted within a container orchestration platform.
12 . The non-transitory machine readable medium of claim 7 , wherein the new or active chain of metadata-only journal records includes an operation header bucket for recording key entries of key records and metadata of the key records and a meta bucket, pointing to the operation header bucket, for recording bucket chain metadata.
13 . The non-transitory machine readable medium of claim 7 , wherein the new or active chain of data journal records includes an operation header bucket for recording key entries of key records and metadata of the key records, a data bucket for recording value entries of value records, and a meta bucket, pointing to the operation header bucket, for recording bucket chain metadata.
14 . The non-transitory machine readable medium of claim 7 , wherein the persistent KV store is stored within a storage device associated with the cluster.
15 . A node of a plurality of nodes of a cluster of a distributed storage system comprising:
one or more processing resources; and
instructions that when executed by the one or more processing resources cause the node to:
maintain an operation log of file system operations that have yet to be flushed to storage of the distributed storage system, including (i) decoupling a plurality of types of journal records from each other by organizing the operation log as a plurality of chains of journal records in which all journal records of a given chain of the plurality of chains of journal records are of a same type of the plurality of types of journal records and (ii) storing the operation log within a persistent key-value (KV) store, wherein the plurality of types of journal records include (a) a data journal record having both data and metadata of a given file system operation at issue and (b) a metadata-only journal record having metadata of the given file system operation at issue;
receive a storage request from a client of the distributed storage system or a file system operable within the node;
determine a type of journal record to be used to log an operation associated with the storage request within the persistent KV store from among the plurality of types of journal records;
based on determining the type of journal record to be used is the metadata-only journal record, log the operation and corresponding metadata within a new or an active chain of metadata-only journal records within the persistent KV store; and
based on determining the type of journal record to be used is the data journal record, log the operation, corresponding data, and the corresponding metadata within a new or an active chain of data journal records within the persistent KV store; and
based on their respective different sets of one or more reclamation dependencies, independently reclaim space in the persistent KV store associated with a plurality of metadata-only journal records and a plurality of data journal records of respective frozen chains of a plurality of frozen chains within the persistent KV store.
16 . The node of claim 15 , wherein the instructions further cause the node to responsive to completion of a local consistency point (CP), issue a call directing the persistent KV store to reclaim the space used by the plurality of metadata-only journal records of a first frozen chain of the plurality of chains associated with the local CP.
17 . The node of claim 15 , wherein the instructions further cause the node to:
after completion of flushing of data and metadata to a remote node of the plurality of nodes from a data journal record of the plurality of data journal records of a second frozen chain of the plurality of frozen chains, determine whether processing of data and metadata of all of the plurality of data journal records has been completed; and
after an affirmative determination, issue a call directing the persistent KV store to reclaim the space used by the plurality of data journal records of the second frozen chain.
18 . The node of claim 15 , wherein the persistent KV store resides within a non-volatile random access memory (NVRAM) of the cluster.
19 . The node of claim 15 , wherein the plurality of nodes are hosted within a container orchestration platform.
20 . The node of claim 15 , wherein the persistent KV store is stored within a storage device associated with the cluster.