Adaptive load balancing in storage system having multiple input-output submission queues
An apparatus comprises a host device configured to communicate over a network with a storage system. The host device comprises a plurality of communication adapters, each of which comprises a corresponding plurality of input-output (IO) submission queues, and a multi-path input-output (MPIO) driver configured to obtain an IO operation that targets a given logical volume of the storage system and to identify a plurality of paths between the host device and the given logical volume. Each identified path comprises a communication adapter and a corresponding IO submission queue of that communication adapter. An IO load weight and a queue depth weight are determined for each identified path. An adaptive weight for each identified path is determined as a function of the determined IO load weight and the determined queue depth weight for that identified path and a target path is selected from the identified paths based at least in part on the determined adaptive weights.
1. An apparatus comprising:
a host device configured to communicate over a network with a storage system comprising a plurality of storage devices;
the host device comprising:
at least one processing device and a plurality of communication adapters, each communication adapter comprising a plurality of input-output submission queues; and
a multi-path input-output driver configured to deliver input-output operations from the host device to the storage system over the network;
wherein the multi-path input-output driver is further configured:
to obtain an input-output operation that targets a given logical volume of the storage system;
to identify a plurality of paths between the host device and the given logical volume, each identified path comprising a corresponding communication adapter of the plurality of communication adapters and a corresponding input-output submission queue of the plurality of input-output submission queues of that corresponding communication adapter;
to determine an input-output load weight for each identified path based at least in part on input-output operations that have been dispatched from the corresponding communication adapter of that identified path to the storage system and are currently in progress;
to determine a queue depth weight for each identified path based at least in part on input-output operations that are pending in the corresponding input-output submission queue of that identified path;
to determine an adaptive weight for each identified path as a mathematical function of the determined input-output load weight and the determined queue depth weight for that identified path;
to select a target path from the identified paths based at least in part on the determined adaptive weight for each identified path; and
to deliver the obtained input-output operation to the given logical volume via the selected target path.
2. The apparatus of claim 1 wherein the input-output load weight for a given identified path is determined based at least in part on a sum of dispatched operation weights of the input-output operations that have been dispatched from the corresponding communication adapter of that identified path to the storage system and are currently in progress.
3. The apparatus of claim 2 wherein the dispatched operation weight of a given input-output operation that has been dispatched from the corresponding communication adapter of the given identified path is determined based at least in part on a type of the given input-output operation.
4. The apparatus of claim 1 wherein the queue depth weight for a given identified path is determined based at least in part on a product of a pending operation weight of each input-output operation that is pending on the corresponding input-output submission queue of the given identified path and the number of input-output operations that are pending on the corresponding input-output submission queue of the given identified path.
5. The apparatus of claim 4 wherein the pending operation weight of a given input-output operation that is pending on the corresponding input-output submission queue of the given identified path is determined based at least in part on a type of the given input-output operation.
6. The apparatus of claim 4 wherein the pending operation weight of a given input-output operation that is pending on the corresponding input-output submission queue of the given identified path is stored in a data structure, each entry in the data structure corresponding to a given communication adapter and one of the corresponding input-output submission queues of the given communication adapter.
7. The apparatus of claim 1 wherein the multi-path input-output driver is further configured to determine a priority of the obtained input-output operation;
wherein each input-output submission queue has a priority level; and
wherein identifying the plurality of paths between the host device and the given logical volume comprises identifying one or more paths that have corresponding input-output submission queues that have a priority level corresponding to the determined priority of the obtained input-output operation.
8. A method comprising:
obtaining, by a multi-path input-output driver of a host device configured to communicate over a network with a storage system comprising a plurality of storage devices, an input-output operation that targets a given logical volume of the storage system;
identifying, by the multi-path input-output driver, a plurality of paths between the host device and the given logical volume, each identified path comprising a corresponding communication adapter of a plurality of communication adapters of the host device and a corresponding input-output submission queue of a plurality of input-output submission queues of that corresponding communication adapter;
determining, by the multi-path input-output driver, an input-output load weight for each identified path based at least in part on input-output operations that have been dispatched from the corresponding communication adapter of that identified path to the storage system and are currently in progress;
determining, by the multi-path input-output driver, a queue depth weight for each identified path based at least in part on input-output operations that are pending in the corresponding input-output submission queue of that identified path;
determining, by the multi-path input-output driver, an adaptive weight for each identified path as a mathematical function of the determined input-output load weight and the determined queue depth weight for that identified path;
selecting, by the multi-path input-output driver, a target path from the identified paths based at least in part on the determined adaptive weight for each identified path;
delivering, by the multi-path input-output driver, the obtained input-output operation to the given logical volume via the selected target path; and
wherein the host device comprises a processor coupled to a memory.
9. The method of claim 8 wherein the input-output load weight for a given identified path is determined based at least in part on a sum of dispatched operation weights of the input-output operations that have been dispatched from the corresponding communication adapter of that identified path to the storage system and are currently in progress.
10. The method of claim 9 wherein the dispatched operation weight of a given input-output operation that has been dispatched from the corresponding communication adapter of the given identified path is determined based at least in part on a type of the given input-output operation.
11. The method of claim 8 wherein the queue depth weight for a given identified path is determined based at least in part on a product of a pending operation weight of each input-output operation that is pending on the corresponding input-output submission queue of the given identified path and the number of input-output operations that are pending on the corresponding input-output submission queue of the given identified path.
12. The method of claim 11 wherein the pending operation weight of a given input-output operation that is pending on the corresponding input-output submission queue of the given identified path is determined based at least in part on a type of the given input-output operation.
13. The method of claim 11 wherein the pending operation weight of a given input-output operation that is pending on the corresponding input-output submission queue of the given identified path is stored in a data structure, each entry in the data structure corresponding to a given communication adapter and one of the corresponding input-output submission queues of the given communication adapter.
14. The method of claim 8 wherein the multi-path input-output driver is further configured to determine a priority of the obtained input-output operation;
wherein each input-output submission queue has a priority level; and
wherein identifying the plurality of paths between the host device and the given logical volume comprises identifying one or more paths that have corresponding input-output submission queues that have a priority level corresponding to the determined priority of the obtained input-output operation.
15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code, when executed by a host device that is configured to communicate over a network with a storage system comprising a plurality of storage devices, the host device comprising at least one processing device and a plurality of communication adapters, each communication adapter comprising a plurality of input-output submission queues, the host device further comprising a multi-path input-output driver configured to deliver input-output operations from the host device to the storage system over the network, causes the multi-path input-output driver:
to obtain an input-output operation that targets a given logical volume of the storage system;
to identify a plurality of paths between the host device and the given logical volume, each identified path comprising a corresponding communication adapter of the plurality of communication adapters and a corresponding input-output submission queue of the plurality of input-output submission queues of that corresponding communication adapter;
to determine an input-output load weight for each identified path based at least in part on input-output operations that have been dispatched from the corresponding communication adapter of that identified path to the storage system and are currently in progress;
to determine a queue depth weight for each identified path based at least in part on input-output operations that are pending in the corresponding input-output submission queue of that identified path;
to determine an adaptive weight for each identified path as a mathematical function of the determined input-output load weight and the determined queue depth weight for that identified path;
to select a target path from the identified paths based at least in part on the determined adaptive weight for each identified path; and
to deliver the obtained input-output operation to the given logical volume via the selected target path.
16. The computer program product of claim 15 wherein the input-output load weight for a given identified path is determined based at least in part on a sum of dispatched operation weights of the input-output operations that have been dispatched from the corresponding communication adapter of that identified path to the storage system and are currently in progress and wherein the dispatched operation weight of a given input-output operation that has been dispatched from the corresponding communication adapter of the given identified path is determined based at least in part on a type of the given input-output operation.
17. The computer program product of claim 15 wherein the queue depth weight for a given identified path is determined based at least in part on a product of a pending operation weight of each input-output operation that is pending on the corresponding input-output submission queue of the given identified path and the number of input-output operations that are pending on the corresponding input-output submission queue of the given identified path.
18. The computer program product of claim 17 wherein the pending operation weight of a given input-output operation that is pending on the corresponding input-output submission queue of the given identified path is determined based at least in part on a type of the given input-output operation.
19. The computer program product of claim 17 wherein the pending operation weight of a given input-output operation that is pending on the corresponding input-output submission queue of the given identified path is stored in a data structure, each entry in the data structure corresponding to a given communication adapter and one of the corresponding input-output submission queues of the given communication adapter.
20. The computer program product of claim 15 wherein program code further causes the multi-path input-output driver to determine a priority of the obtained input-output operation;
wherein each input-output submission queue has a priority level; and
wherein identifying the plurality of paths between the host device and the given logical volume comprises identifying one or more paths that have corresponding input-output submission queues that have a priority level corresponding to the determined priority of the obtained input-output operation.