Metadata update journal destaging with preload phase for efficient metadata recovery in a distributed storage system
At least one processing device is configured to detect a failure event impacting at least a first storage node of a distributed storage system, and responsive to the detected failure event, to initiate a metadata recovery process for at least the first storage node. In conjunction with the metadata recovery process, destaging of a metadata update journal of the first storage node is performed, the destaging of the metadata update journal of the first storage node being performed in multiple phases, including at least a preload phase in which, for each of a plurality of pages required for the destaging of the metadata update journal, one or more address locks are obtained for the page, the page is preloaded into a memory of the first storage node from persistent storage accessible to the first storage node, and the one or more address locks are released, and an update and write phase.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
said at least one processing device being configured:
to detect a failure event impacting at least a first storage node of a plurality of storage nodes of a distributed storage system;
responsive to the detected failure event, to initiate a metadata recovery process for at least the first storage node; and
in conjunction with the metadata recovery process, to perform destaging of a metadata update journal of the first storage node;
the destaging of the metadata update journal of the first storage node being performed in multiple phases, including at least:
a preload phase in which, for each of a plurality of pages required for the destaging of the metadata update journal, one or more address locks are obtained for the page, the page is preloaded into a memory of the first storage node from persistent storage accessible to the first storage node, and the one or more address locks are released; and
an update and write phase in which a plurality of address locks are obtained for the preloaded pages, metadata updates are performed in the first storage node using the preloaded pages, corresponding updated pages are written to the persistent storage, and the address locks are released.
2. The apparatus of claim 1 wherein said at least one processing device comprises at least a portion of a distributed storage controller of the distributed storage system.
3. The apparatus of claim 1 wherein the storage nodes of the distributed storage system are interconnected in a mesh network.
4. The apparatus of claim 1 wherein each of at least a subset of the storage nodes of the distributed storage system comprises a set of processing modules configured to communicate with corresponding sets of processing modules on other ones of the storage nodes, a given such set of processing modules comprising at least a routing module, a control module and a data module, the sets of processing modules of the storage nodes of the distributed storage system collectively comprising at least a portion of a distributed storage controller of the distributed storage system.
5. The apparatus of claim 1 wherein the detected failure event comprises at least one of: (i) a failure of at least one process executing on the first storage node, and (ii) a failure of the first storage node itself.
6. The apparatus of claim 1 wherein initiating a metadata recovery process for at least the first storage node comprises initiating the metadata recovery process on the first storage node as a background process performed while the first storage node processes input-output operations from one or more host devices.
7. The apparatus of claim 1 wherein the pages comprise respective fixed-size pages arranged into page groups.
8. The apparatus of claim 7 wherein for a given one of the pages required for destaging of the metadata update journal, obtaining one or more address locks in the preload phase for the given data page comprises obtaining an address lock for the given data page and an address lock for its corresponding page group.
9. The apparatus of claim 7 wherein obtaining address locks for the preloaded pages in the update and write phase comprises:
obtaining address locks for respective ones of the preloaded pages; and
obtaining address locks for respective page groups corresponding to the preloaded pages.
10. The apparatus of claim 7 wherein the updated pages are written to the persistent storage in batches with each such batch comprising all of the updated pages of one or more of the page groups.
11. The apparatus of claim 1 wherein responsive to completion of the metadata update journal destaging, one or more subsequent iterations of metadata update journal destaging are performed each using the update and write phase without the preload phase.
12. The apparatus of claim 1 wherein the phases are configured to decouple the preloading of pages from persistent storage and the updating and writing of those preloaded pages so as to thereby limit an amount of time for which an input-output operation from a host device that requires a given one of the pages is delayed waiting for release of an address lock on the given page.
13. The apparatus of claim 1 wherein the persistent storage comprises a first plurality of storage devices associated with the first storage node and one or more additional pluralities of storage devices associated with respective additional ones of the storage nodes.
14. The apparatus of claim 13 wherein the first plurality of storage devices are implemented in a storage array enclosure of the first storage node.
15. A method comprising:
detecting a failure event impacting at least a first storage node of a plurality of storage nodes of a distributed storage system;
responsive to the detected failure event, initiating a metadata recovery process for at least the first storage node; and
in conjunction with the metadata recovery process, performing destaging of a metadata update journal of the first storage node;
the destaging of the metadata update journal of the first storage node being performed in multiple phases, including at least:
a preload phase in which, for each of a plurality of pages required for the destaging of the metadata update journal, one or more address locks are obtained for the page, the page is preloaded into a memory of the first storage node from persistent storage accessible to the first storage node, and the one or more address locks are released; and
an update and write phase in which a plurality of address locks are obtained for the preloaded pages, metadata updates are performed in the first storage node using the preloaded pages, corresponding updated pages are written to the persistent storage, and the address locks are released;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
16. The method of claim 15 wherein initiating a metadata recovery process for at least the first storage node comprises initiating the metadata recovery process on the first storage node as a background process performed while the first storage node processes input-output operations from one or more host devices.
17. The method of claim 15 wherein the pages comprise respective fixed-size pages arranged into page groups, and wherein for a given one of the pages required for destaging of the metadata update journal, obtaining one or more address locks in the preload phase for the given data page comprises obtaining an address lock for the given data page and an address lock for its corresponding page group.
18. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to detect a failure event impacting at least a first storage node of a plurality of storage nodes of a distributed storage system;
responsive to the detected failure event, to initiate a metadata recovery process for at least the first storage node; and
in conjunction with the metadata recovery process, to perform destaging of a metadata update journal of the first storage node;
the destaging of the metadata update journal of the first storage node being performed in multiple phases, including at least:
a preload phase in which, for each of a plurality of pages required for the destaging of the metadata update journal, one or more address locks are obtained for the page, the page is preloaded into a memory of the first storage node from persistent storage accessible to the first storage node, and the one or more address locks are released; and
an update and write phase in which a plurality of address locks are obtained for the preloaded pages, metadata updates are performed in the first storage node using the preloaded pages, corresponding updated pages are written to the persistent storage, and the address locks are released.
19. The computer program product of claim 18 wherein initiating a metadata recovery process for at least the first storage node comprises initiating the metadata recovery process on the first storage node as a background process performed while the first storage node processes input-output operations from one or more host devices.
20. The computer program product of claim 18 wherein the pages comprise respective fixed-size pages arranged into page groups, and wherein for a given one of the pages required for destaging of the metadata update journal, obtaining one or more address locks in the preload phase for the given data page comprises obtaining an address lock for the given data page and an address lock for its corresponding page group.