Coalescing metadata for mirroring to a remote storage node in a cluster storage system
Described herein are a system and method for remote mirroring/copying data and metadata sets from a local node to a remote node that reduces the number of metadata sets that are mirrored. In some embodiments, the local node may coalesce metadata sets into metadata chains, each metadata chain comprising a grouping of two or more metadata sets. In some instances, a “representative” metadata set of a metadata chain may be selected for sending to the remote node for storing, wherein the other metadata sets of the metadata chain are not sent to the remote node. In these embodiments, the selected metadata set may represent all the metadata sets in the chain and be the only metadata set in the chain that is transmitted and stored to the remote node. As such, the network congestion between the local and remote nodes may be reduced.
1. A computing device, comprising:
a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of facilitating storage system mirroring and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:
receive a plurality of data and metadata sets from a client, each received data or metadata set comprising a data or metadata request from the client to store the data or metadata set to a remote node;
produce a plurality of metadata chains, each metadata chain comprising a grouping of at least two received metadata requests wherein at least one of the metadata chains comprises at least a new received metadata request and a previous received metadata request to be stored to a same storage address location at the remote node;
store each received metadata request to a metadata request (MR) data structure according to the time order the requests are received, each metadata chain comprising a head metadata request comprising an earliest received metadata request in the metadata chain and a tail metadata request comprising a last received metadata request in the metadata chain; and
transmit a representative metadata request that comprises the tail metadata in the metadata chain to the remote node for storage.
2. The system of claim 1 , wherein the processor is further configured to execute the machine executable code to cause the processor to transmit each received data request to the remote node for storage.
3. The computing device of claim 1 , wherein:
the metadata sets specify a total accumulated number of data sets produced by the client; and
the data contained in the representative metadata request encompasses the data contained in the other metadata sets of the metadata chain.
4. The system of claim 1 , wherein:
the previous metadata request has not yet been transmitted to the remote node.
5. The computing device of claim 1 , wherein the new received metadata request is grouped with the previous received metadata request when both metadata requests have the same data size.
6. The system of claim 4 , wherein:
both metadata requests have the same data size.
7. A method, comprising:
receiving a plurality of data and metadata sets from a client, each received data or metadata set comprising a data or metadata request from the client to store the data or metadata set to a remote node;
producing a plurality of metadata chains, each metadata chain comprising a grouping of at least two received metadata requests wherein at least one of the metadata chains comprises at least a new received metadata request and a previous received metadata request to be stored to a same storage address location at the remote node;
storing each received metadata request to a metadata request (MR) data structure according to the time order the requests are received, each metadata chain comprising a head metadata request comprising an earliest received metadata request in the metadata chain and a tail metadata request comprising the last received metadata request in the metadata chain; and
transmitting a representative metadata request that comprises the tail metadata in the metadata chain to the remote node for storage.
8. The method of claim 7 , further comprising:
transmitting each received data request to the remote node for storage.
9. The method of claim 7 , wherein:
the metadata sets specify a total accumulated number of data sets produced by the client; and
the data contained in the representative metadata request encompasses the data contained in the other metadata sets of the metadata chain.
10. The method of claim 7 , wherein:
the previous metadata request has not yet been transmitted to the remote node.
11. The method of claim 7 , wherein the new received metadata request is grouped with the previous received metadata request when both metadata requests have the same data size.
12. The method of claim 10 , wherein:
both metadata requests have the same data size.
13. A non-transitory computer readable medium having stored thereon instructions for performing a method of facilitating storage system mirroring, which when executed by a at least one machine, causes the machine to:
receive a plurality of data and metadata sets from a client, each received data or metadata set comprising a data or metadata request from the client to store the data or metadata set to a remote node;
produce a plurality of metadata chains, each metadata chain comprising a grouping of at least two received metadata requests wherein at least one of the metadata chains comprises at least a new received metadata request and a previous received metadata request to be stored to a same storage address location at the remote node;
store each received metadata request to a metadata request (MR) data structure according to the time order the requests are received, each metadata chain comprising a head metadata request comprising an earliest received metadata request in the metadata chain and a tail metadata request comprising a last received metadata request in the metadata chain; and
transmit a representative metadata request that comprises the tail metadata in the metadata chain to the remote node for storage.
14. The non-transitory computer readable medium of claim 13 , wherein the instructions, when executed by the at least one machine, further cause the machine to transmit each received data request to the remote node for storage.
15. The non-transitory computer readable medium of claim 13 , wherein:
the metadata sets specify a total accumulated number of data sets produced by the client; and
the data contained in the representative metadata request encompasses the data contained in the other metadata sets of the metadata chain.
16. The non-transitory computer readable medium of claim 13 , wherein:
the previous metadata request has not yet been transmitted to the remote node.
17. The non-transitory computer readable medium of claim 13 , wherein the new received metadata request is grouped with the previous received metadata request when both metadata requests have the same data size.
18. The non-transitory computer readable medium of claim 16 , wherein:
both metadata requests have the same data size.