Recovery flow with reduced address lock contention in a content addressable storage system
An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory, with the processing device being configured to detect a failure of at least one storage node that impacts a write cache destaging process in a distributed content addressable storage (CAS) system comprising a plurality of storage nodes each associated with one or more storage devices. Responsive to the detected failure, the processing device issues one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process, thereby illustratively reducing address lock contention in the CAS system. The processing device illustratively comprises at least a portion of a distributed storage controller of the CAS system. Other embodiments include methods and computer program products.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
said at least one processing device being configured:
to detect a failure of at least one storage node that impacts a write cache destaging process in a distributed content addressable storage system comprising a plurality of storage nodes each associated with one or more storage devices; and
responsive to the detected failure, to issue one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process;
wherein detecting a failure of at least one storage node that impacts a write cache destaging process comprises detecting the failure of a particular storage node prior to acquisition of address locks for respective ones of one or more data pages targeted by one or more write requests to be destaged from a write cache of the distributed content addressable storage system as part of the write cache destaging process;
wherein issuing one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process comprises issuing the one or more write cache metadata preload commands from a first one of the storage nodes to at least a second one of the storage nodes; and
wherein the write cache destaging process is delayed until each of the one or more write cache metadata preload commands issued for a designated write cache destage batch is acknowledged as complete.
2. The apparatus of claim 1 wherein said at least one processing device comprises at least a portion of a storage controller of the distributed content addressable storage system.
3. The apparatus of claim 1 wherein the storage nodes of the distributed content addressable storage system are interconnected in a mesh network.
4. The apparatus of claim 1 wherein each of at least a subset of the storage nodes of the distributed content addressable storage system comprises a set of processing modules configured to communicate with corresponding sets of processing modules on other ones of the storage nodes, a given such set of processing modules comprising at least a routing module, a control module and a data module, the sets of processing modules of the storage nodes of the distributed content addressable storage system collectively comprising at least a portion of a storage controller of the distributed content addressable storage system.
5. The apparatus of claim 1 wherein address locks are placed on respective ones of a plurality of data pages targeted by one or more write requests to be destaged from a write cache of the distributed content addressable storage system as part of the write cache destaging process.
6. The apparatus of claim 1 wherein detecting a failure of at least one storage node that impacts a write cache destaging process comprises detecting a failure of a data module of a particular storage node that is associated with one or more storage devices that store one or more data pages targeted by one or more write requests to be destaged from a write cache of the distributed content addressable storage system as part of the write cache destaging process.
7. The apparatus of claim 1 wherein detecting a failure of at least one storage node that impacts a write cache destaging process comprises detecting the failure of a particular storage node prior to initiation of the write cache destaging process.
8. The apparatus of claim 1 wherein a plurality of write cache metadata preload commands are issued from a control module of the first storage node to respective ones of a plurality of data modules of respective ones of the second storage node and one or more additional ones of the storage nodes.
9. The apparatus of claim 1 wherein each of the one or more write cache metadata preload commands comprises a remote procedure call (RPC).
10. The apparatus of claim 1 wherein each of the one or more write cache metadata preload commands is issued in conjunction with an initial step of the write cache destaging process.
11. The apparatus of claim 1 wherein issuing one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process comprises issuing multiple write cache metadata preload commands to respective ones of a plurality of data modules that are associated with a designated write cache destage batch of the write cache destaging process.
12. A method comprising:
detecting a failure of at least one storage node that impacts a write cache destaging process in a distributed content addressable storage system comprising a plurality of storage nodes each associated with one or more storage devices; and
responsive to the detected failure, issuing one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process;
wherein detecting a failure of at least one storage node that impacts a write cache destaging process comprises detecting the failure of a particular storage node prior to acquisition of address locks for respective ones of one or more data pages targeted by one or more write requests to be destaged from a write cache of the distributed content addressable storage system as part of the write cache destaging process;
wherein issuing one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process comprises issuing the one or more write cache metadata preload commands from a first one of the storage nodes to at least a second one of the storage nodes;
wherein the write cache destaging process is delayed until each of the one or more write cache metadata preload commands issued for a designated write cache destage batch is acknowledged as complete; and
wherein the method is implemented by at least one processing device comprising a processor coupled to a memory.
13. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to detect a failure of at least one storage node that impacts a write cache destaging process in a distributed content addressable storage system comprising a plurality of storage nodes each associated with one or more storage devices; and
responsive to the detected failure, to issue one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process;
wherein detecting a failure of at least one storage node that impacts a write cache destaging process comprises detecting the failure of a particular storage node prior to acquisition of address locks for respective ones of one or more data pages targeted by one or more write requests to be destaged from a write cache of the distributed content addressable storage system as part of the write cache destaging process;
wherein issuing one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process comprises issuing the one or more write cache metadata preload commands from a first one of the storage nodes to at least a second one of the storage nodes; and
wherein the write cache destaging process is delayed until each of the one or more write cache metadata preload commands issued for a designated write cache destage batch is acknowledged as complete.
14. The computer program product of claim 13 wherein a plurality of write cache metadata preload commands are issued from a control module of the first storage node to respective ones of a plurality of data modules of respective ones of the second storage node and one or more additional ones of the storage nodes.
15. The computer program product of claim 13 wherein each of the one or more write cache metadata preload commands is issued in conjunction with an initial step of the write cache destaging process.
16. The computer program product of claim 13 wherein issuing one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process comprises issuing multiple write cache metadata preload commands to respective ones of a plurality of data modules that are associated with a designated write cache destage batch of the write cache destaging process.
17. The method of claim 12 wherein a plurality of write cache metadata preload commands are issued from a control module of the first storage node to respective ones of a plurality of data modules of respective ones of the second storage node and one or more additional ones of the storage nodes.
18. The method of claim 12 wherein each of the one or more write cache metadata preload commands is issued in conjunction with an initial step of the write cache destaging process.
19. The method of claim 12 wherein issuing one or more write cache metadata preload commands to direct one or more other ones of the storage nodes to preload from their associated storage devices metadata characterizing one or more data pages that are subject to the write cache destaging process comprises issuing multiple write cache metadata preload commands to respective ones of a plurality of data modules that are associated with a designated write cache destage batch of the write cache destaging process.
20. The method of claim 12 wherein each of the one or more write cache metadata preload commands comprises a remote procedure call (RPC).