Dynamic modification of IO shaping mechanisms of multiple storage nodes in a distributed storage system
At least one processing device is configured to detect a failure event impacting at least a first storage node of a plurality of storage nodes of a distributed storage system, and responsive to the detected failure event, to modify an input-output (IO) shaping mechanism in each of the storage nodes in order to at least temporarily reduce a total number of IO operations that are concurrently processed in the distributed storage system. For example, modifying an IO shaping mechanism in each of the storage nodes illustratively comprises transitioning the IO shaping mechanism in each of the storage nodes from a first operating mode to a second operating mode that is different than the first operating mode. The second operating mode of the IO shaping mechanism illustratively has a relatively faster responsiveness to changes in IO operation latency as compared to the first operating mode of the IO shaping mechanism.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
said at least one processing device being configured:
to detect a failure event impacting at least a first storage node of a plurality of storage nodes of a distributed storage system; and
responsive to the detected failure event, to modify an input-output shaping mechanism in each of the storage nodes in order to at least temporarily reduce a total number of input-output operations that are concurrently processed in the distributed storage system;
wherein modifying an input-output shaping mechanism in each of the storage nodes comprises transitioning the input-output shaping mechanism in each of the storage nodes from a first operating mode to a second operating mode that is different than the first operating mode; and
wherein the second operating mode of the input-output shaping mechanism has a relatively faster responsiveness to changes in input-output operation latency as compared to the first operating mode of the input-output shaping mechanism.
2. The apparatus of claim 1 wherein the detected failure event comprises at least one of: (i) a failure of at least one process executing on the first storage node, (ii) a failure of one or more storage devices of the first storage node, and (iii) a failure of the first storage node itself.
3. The apparatus of claim 1 wherein the input-output shaping mechanism in the first operating mode controls processing of input-output operations in order to satisfy one or more constraints relating to average latency over a plurality of input-output operations.
4. The apparatus of claim 1 wherein the input-output shaping mechanism in the second operating mode controls processing of input-output operations in order to satisfy one or more constraints relating to latency of individual input-output operations.
5. The apparatus of claim 1 wherein the input-output shaping mechanism in a given one of the storage nodes is configured to measure input-output operation latency and to adjust a number of parallel input-output operations processed by the given storage node responsive to the measurement of input-output operation latency.
6. The apparatus of claim 1 wherein modifying an input-output shaping mechanism in each of the storage nodes further comprises:
identifying a type of the detected failure event; and
providing a particular one of a plurality of available modifications of the input-output shaping mechanism in a given one of the storage nodes based at least in part on the identified type of the detected failure event.
7. The apparatus of claim 6 wherein different ones of the plurality of available modifications of the input-output shaping mechanism provide different reductions in number of parallel input-output operations processed by the given storage node.
8. The apparatus of claim 1 wherein each of at least a subset of the storage nodes of the distributed storage system comprises a set of processing modules configured to communicate with corresponding sets of processing modules on other ones of the storage nodes, a given such set of processing modules comprising at least a routing module, a control module and a data module, the sets of processing modules of the storage nodes of the distributed storage system collectively comprising at least a portion of a distributed storage controller of the distributed storage system.
9. The apparatus of claim 1 wherein modifying operation of an input-output shaping mechanism in each of the storage nodes comprises:
generating a backpressure hint; and
sending the backpressure hint to each of the storage nodes;
wherein the input-output shaping mechanism is modified in each of the storage nodes responsive to receipt of the backpressure hint.
10. The apparatus of claim 9 wherein generating the backpressure hint and sending the backpressure hint to each of the storage nodes are performed by a system manager of the distributed storage system.
11. The apparatus of claim 9 wherein sending the backpressure hint to each of the storage nodes comprises sending the backpressure hint from a system manager of the distributed storage system over a dedicated channel to each of the storage nodes.
12. The apparatus of claim 9 wherein a given one of the storage nodes in response to receipt of the backpressure hint modifies its input-output shaping mechanism to effectively reduce a number of parallel input-output operations processed by the given storage node.
13. A method comprising:
detecting a failure event impacting at least a first storage node of a plurality of storage nodes of a distributed storage system; and
responsive to the detected failure event, modifying an input-output shaping mechanism in each of the storage nodes in order to at least temporarily reduce a total number of input-output operations that are concurrently processed in the distributed storage system;
wherein modifying an input-output shaping mechanism in each of the storage nodes comprises transitioning the input-output shaping mechanism in each of the storage nodes from a first operating mode to a second operating mode that is different than the first operating mode;
wherein the second operating mode of the input-output shaping mechanism has a relatively faster responsiveness to changes in input-output operation latency as compared to the first operating mode of the input-output shaping mechanism; and
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
14. The method of claim 13 wherein modifying operation of an input-output shaping mechanism in each of the storage nodes comprises:
generating a backpressure hint; and
sending the backpressure hint to each of the storage nodes;
wherein the input-output shaping mechanism is modified in each of the storage nodes responsive to receipt of the backpressure hint; and
wherein a given one of the storage nodes in response to receipt of the backpressure hint modifies its input-output shaping mechanism to effectively reduce a number of parallel input-output operations processed by the given storage node.
15. The method of claim 14 wherein generating the backpressure hint and sending the backpressure hint to each of the storage nodes are performed by a system manager of the distributed storage system.
16. The method of claim 14 wherein sending the backpressure hint to each of the storage nodes comprises sending the backpressure hint from a system manager of the distributed storage system over a dedicated channel to each of the storage nodes.
17. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to detect a failure event impacting at least a first storage node of a plurality of storage nodes of a distributed storage system; and
responsive to the detected failure event, to modify an input-output shaping mechanism in each of the storage nodes in order to at least temporarily reduce a total number of input-output operations that are concurrently processed in the distributed storage system;
wherein modifying an input-output shaping mechanism in each of the storage nodes comprises transitioning the input-output shaping mechanism in each of the storage nodes from a first operating mode to a second operating mode that is different than the first operating mode; and
wherein the second operating mode of the input-output shaping mechanism has a relatively faster responsiveness to changes in input-output operation latency as compared to the first operating mode of the input-output shaping mechanism.
18. The computer program product of claim 17 wherein modifying operation of an input-output shaping mechanism in each of the storage nodes comprises:
generating a backpressure hint; and
sending the backpressure hint to each of the storage nodes;
wherein the input-output shaping mechanism is modified in each of the storage nodes responsive to receipt of the backpressure hint; and
wherein a given one of the storage nodes in response to receipt of the backpressure hint modifies its input-output shaping mechanism to effectively reduce a number of parallel input-output operations processed by the given storage node.
19. The computer program product of claim 18 wherein generating the backpressure hint and sending the backpressure hint to each of the storage nodes are performed by a system manager of the distributed storage system.
20. The computer program product of claim 18 wherein sending the backpressure hint to each of the storage nodes comprises sending the backpressure hint from a system manager of the distributed storage system over a dedicated channel to each of the storage nodes.