Detection and mitigation of slow drain issues using response times and storage-side latency view
An apparatus in one embodiment comprises at least one processing device configured to control delivery of input-output (IO) operations from a host device to a storage system over selected ones of a plurality of paths through a network, and to monitor response times for particular ones of the IO operations sent from the host device to the storage system. The at least one processing device is further configured to interact with the storage system to determine network latency from a viewpoint of the storage system, and responsive to (i) at least a subset of the monitored response times being above a first threshold and (ii) the network latency from the viewpoint of the storage system being above a second threshold, to at least temporarily modify a manner in which additional ones of the IO operations are sent from the host device to the storage system.
1. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
wherein the at least one processing device is configured:
to control delivery of input-output operations from a host device to a storage system over selected ones of a plurality of paths through a network;
to monitor response times for particular ones of the input-output operations sent from the host device to the storage system;
to interact with the storage system to determine network latency from a viewpoint of the storage system; and
responsive to (i) at least a subset of the monitored response times being above a first threshold and (ii) the network latency from the viewpoint of the storage system being above a second threshold, to at least temporarily modify a manner in which additional ones of the input-output operations are sent from the host device to the storage system.
2. The apparatus of claim 1 wherein the at least one processing device comprises at least a portion of the host device.
3. The apparatus of claim 1 wherein the selected ones of the plurality of paths are associated with respective initiator-target pairs, the initiators being implemented on the host device and the targets being implemented on the storage system, the initiators of the initiator-target pairs comprising respective host bus adaptors of the host device and the targets of the initiator-target pairs comprising respective storage array ports of the storage system.
4. The apparatus of claim 1 wherein said at least one processing device comprises a multi-path input-output driver of the host device, with the multi-path input-output driver of the host device being configured to control the delivery of the input-output operations from the host device to the storage system over the selected ones of the plurality of paths through the network, and to perform at least a portion of the monitoring of the response times, the interacting with the storage system to determine the network latency from the viewpoint of the storage system, and the at least temporarily modifying of the manner in which additional ones of the input-output operations are sent from the host device to the storage system.
5. The apparatus of claim 1 wherein at least temporarily modifying a manner in which additional ones of the input-output operations are sent from the host device to the storage system comprises at least temporarily reducing a rate at which additional ones of the input-output operations are sent from the host device to the storage system.
6. The apparatus of claim 1 wherein at least temporarily modifying a manner in which additional ones of the input-output operations are sent from the host device to the storage system comprises at least temporarily utilizing one or more alternative paths for sending additional ones of the input-output operations from the host device to the storage system.
7. The apparatus of claim 1 wherein interacting with the storage system to determine network latency from a viewpoint of the storage system comprises:
sending a designated command from the host device to the storage system;
receiving a response to the designated command; and
determining the network latency based at least in part on the received response;
wherein the designated command comprises a command for which the storage system, responsive to recognition of the command, directly returns the response without further processing of the command in the storage system.
8. The apparatus of claim 7 wherein the designated command comprises a ping command.
9. The apparatus of claim 7 wherein the designated command comprises a vendor unique command of a storage access protocol utilized by the host device to access the storage system over the network.
10. The apparatus of claim 1 wherein monitoring response times for particular ones of the input-output operations sent from the host device to the storage system comprises monitoring response times for the particular ones of the input-output operations on at least one of a per-path basis for each of at least a subset of the plurality of paths and on a per-device basis for each of a plurality of logical storage devices of the storage system.
11. The apparatus of claim 1 wherein the at least one processing device is further configured to perform one or more checks responsive to (i) at least a subset of the monitored response times being above the first threshold and (ii) the network latency from the viewpoint of the storage system being above the second threshold, and wherein the at least temporary modification of the manner in which additional ones of the input-output operations are sent from the host device to the storage system is determined based at least in part on at least one result of the one or more checks.
12. The apparatus of claim 1 wherein a given one of the one or more checks performed by the at least one processing device comprises determining whether or not one or more of the plurality of paths each exhibits at least a threshold amount of mismatch between a negotiated rate of its initiator and a negotiated rate of its target.
13. The apparatus of claim 1 wherein a given one of the one or more checks performed by the at least one processing device comprises determining whether or not one or more fabric performance impact notifications have been received by the host device for one or more of the plurality of paths.
14. The apparatus of claim 1 wherein said at least one processing device is further configured to at least partially reverse the modification of the manner in which additional ones of the input-output operations are sent from the host device to the storage system.
15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code, when executed by at least one processing device comprising a processor coupled to a memory, causes the at least one processing device:
to control delivery of input-output operations from a host device to a storage system over selected ones of a plurality of paths through a network;
to monitor response times for particular ones of the input-output operations sent from the host device to the storage system;
to interact with the storage system to determine network latency from a viewpoint of the storage system; and
responsive to (i) at least a subset of the monitored response times being above a first threshold and (ii) the network latency from the viewpoint of the storage system being above a second threshold, to at least temporarily modify a manner in which additional ones of the input-output operations are sent from the host device to the storage system.
16. The computer program product of claim 15 wherein interacting with the storage system to determine network latency from a viewpoint of the storage system comprises:
sending a designated command from the host device to the storage system;
receiving a response to the designated command; and
determining the network latency based at least in part on the received response;
wherein the designated command comprises a command for which the storage system, responsive to recognition of the command, directly returns the response without further processing of the command in the storage system.
17. The computer program product of claim 15 wherein the at least one processing device is further configured to perform one or more checks responsive to (i) at least a subset of the monitored response times being above the first threshold and (ii) the network latency from the viewpoint of the storage system being above the second threshold, and wherein the at least temporary modification of the manner in which additional ones of the input-output operations are sent from the host device to the storage system is determined based at least in part on at least one result of the one or more checks.
18. A method comprising:
controlling delivery of input-output operations from a host device to a storage system over selected ones of a plurality of paths through a network;
monitoring response times for particular ones of the input-output operations sent from the host device to the storage system;
interacting with the storage system to determine network latency from a viewpoint of the storage system; and
responsive to (i) at least a subset of the monitored response times being above a first threshold and (ii) the network latency from the viewpoint of the storage system being above a second threshold, to at least temporarily modifying a manner in which additional ones of the input-output operations are sent from the host device to the storage system.
19. The method of claim 18 wherein interacting with the storage system to determine network latency from a viewpoint of the storage system comprises:
sending a designated command from the host device to the storage system;
receiving a response to the designated command; and
determining the network latency based at least in part on the received response;
wherein the designated command comprises a command for which the storage system, responsive to recognition of the command, directly returns the response without further processing of the command in the storage system.
20. The method of claim 18 further comprising performing one or more checks responsive to (i) at least a subset of the monitored response times being above the first threshold and (ii) the network latency from the viewpoint of the storage system being above the second threshold, and wherein the at least temporary modification of the manner in which additional ones of the input-output operations are sent from the host device to the storage system is determined based at least in part on at least one result of the one or more checks.