Data storage system with port congestion hinting for host multipathing
A data processing system includes host computers and a data storage system that (1) periodically calculates port congestion values for storage system ports based on per-port values for (a) a number of storage IO commands being processed by each port and (b) CPU utilization of one or more CPUs assigned to each port, and (2) periodically sends the port congestion values to the host computers. Each host computer includes multipathing logic for selectively directing storage IO commands on selected paths to the data storage system. The multipathing logic receives the port congestion values from the data storage system and utilizes the port congestion values for path selection by preferentially directing first storage IO commands of higher service level objective to storage system ports having lower congestion and preferentially directing second storage IO commands of lower service level objective to storage system ports having higher congestion.
1. A data processing system, comprising:
a plurality of host computers executing respective applications with respective storage IO workloads, the host computers including respective host ports; and
a data storage system having storage system ports coupled to the host ports of the host computers to receive storage IO commands therefrom and to send storage IO responses thereto,
the data storage system being configured and operative to (1) periodically calculate port congestion values for the storage system ports based on per-port values for (a) a number of storage IO commands being processed by each port and (b) CPU utilization of one or more CPUs assigned to each port, and (2) periodically send the port congestion values to the host computers,
each host computer including respective multipathing logic for selectively directing storage IO commands of the host computer on selected paths to the data storage system, each path including a pairing of a respective host port and respective storage system port, the multipathing logic being configured and operative to receive the port congestion values from the data storage system and utilize the port congestion values for path selection by preferentially directing first storage IO commands of higher service level objective to storage system ports having lower congestion and preferentially directing second storage IO commands of lower service level objective to storage system ports having higher congestion.
2. The data processing system of claim 1 , further including a fabric containing the paths on which the storage IO commands and corresponding responses are sent between the host computers and the data storage system, and wherein:
the first storage IO commands of higher service level objective are storage IO commands of a first-tier application executing on a first host computer, and the second storage IO commands of lower service level objective are storage IO commands of a lower-tier application executing on a second host computer, the lower-tier application having a respective workload that would, in the absence of the preferential directing of the first and second storage IO commands, consume a larger share of the fabric than a respective workload of the first-tier application.
3. The data processing system of claim 1 , operating in a virtualized environment in which the host computers host virtual machines whose storage IO commands have respective service level objectives, the host computers using an application programming interface for storage awareness to make the multipathing logic aware of the service level objectives of the storage IO commands.
4. The data processing system of claim 3 , a host computer using an application programming interface obtains a list of virtual machines on the host computer which are consuming storage of a particular service level objective.
5. The data processing system of claim 1 , wherein the host computers send a vendor SCSI command to the data storage system to obtain the port congestion values.
6. The data processing system of claim 5 , wherein the vendor SCSI command is either a first type that requests a specific port's congestion value and is repeated for each port, or a second type requesting congestion values for all the ports visible to the host computer, the vendor SCSI command being sent in a periodic manner and the returned data being updated into an internal data structure of the multipathing logic.
7. The data processing system of claim 1 , wherein the data storage system includes front end emulations through which the host computers connect to storage resources, the front end emulations employing executor threads to provide a major part of front end processing of the storage IO commands.
8. The data processing system of claim 7 , wherein the front end emulations are configured and operative to process each received storage IO command by (1) allocating a respective job record including a memory buffer that stores information about the storage IO command, (2) selecting an executor thread and assigning it to further process the storage IO command, the executor thread being chosen to evenly load the set of executor threads of the front end emulation.
9. The data processing system of claim 1 , wherein the port congestion values are calculated based on CPU utilization and number of active jobs in each port.
10. The data processing system of claim 9 , wherein each port has a set of allocated CPU cores and the CPU utilization part of the port congestion value calculation includes determining idle time of allocated CPU cores, with higher idle times being interpreted as less CPU utilization.
11. The data processing system of claim 10 , wherein the port congestion value calculation includes:
in a first step, calculating an average percentage of CPU idle time for each CPU core;
in a second step, calculating an average percentage of CPU idle time for each of a set of front end emulations using the CPU cores;
in a third step, sorting the average percentages of CPU idle time in descending order;
in a fourth step, calculating a rank for each of the front end emulations from the sorted average percentages;
in a fifth step, calculating an average base congestion rate per port based on the ranks of the front end emulations;
in a sixth step, calculating an average number of jobs per port; and
in a seventh step, calculating an average port congestion rate from the average base congestion rate and average number of jobs per port.
12. The data processing system of claim 1 , wherein the host computers are configured and operative to (a) monitor respective latencies of the first and second storage IO commands, and (b) refresh the port congestion values from the data storage system upon the latencies for the first storage IO commands becoming greater than the latencies for the second storage IO commands.
13. The data processing system of claim 1 , wherein the multipathing logic is operative:
(1) for the higher service level objective and every eligible path, (a) obtain information about the least congested port(s) and the paths associated therewith, (b) calculate a current number of pending read and write storage IO commands, (c) calculate respective path weights based on the calculated values, and (d) selecting a path with a lowest weight for each first storage IO command; and
(2) for the lower service level objective and every eligible path, (a) obtain information about the more congested port(s) and the paths associated therewith, (b) calculate a current number of pending read and write storage IO commands, and (c) selecting a path with a highest congestion value for each second storage IO command.
14. A method of operating a data processing system having a plurality of host computers and a data storage system, the host computers including respective host ports and executing respective applications with respective storage IO workloads, the data storage system having storage system ports coupled to the host ports of the host computers to receive storage IO commands therefrom and to send storage IO responses thereto, comprising:
by the data storage system, (1) periodically calculating port congestion values for the storage system ports based on per-port values for (a) a number of storage IO commands being processed by each port and (b) CPU utilization of one or more CPUs assigned to each port, and (2) periodically sending the port congestion values to the host computers; and
by respective multipathing logic of each of the host computers, for selectively directing storage IO commands of the host computer on selected paths to the data storage system, each path including a pairing of a respective host port and respective storage system port, (1) receiving the port congestion values from the data storage system, and (2) utilizing the port congestion values for path selection by preferentially directing first storage IO commands of higher service level objective to storage system ports having lower congestion and preferentially directing second storage IO commands of lower service level objective to storage system ports having higher congestion.
15. The method of claim 14 , wherein the data processing system further includes a fabric containing the paths on which the storage IO commands and corresponding responses are sent between the host computers and the data storage system, and wherein:
the first storage IO commands of higher service level objective are storage IO commands of a first-tier application executing on a first host computer, and the second storage IO commands of lower service level objective are storage IO commands of a lower-tier application executing on a second host computer, the lower-tier application having a respective workload that would, in the absence of the preferential directing of the first and second storage IO commands, consume a larger share of the fabric than a respective workload of the first-tier application.
16. The method of claim 14 , operating in a virtualized environment in which the host computers host virtual machines whose storage IO commands have respective service level objectives, the host computers using an application programming interface for storage awareness to make the multipathing logic aware of the service level objectives of the storage IO commands.
17. The method of claim 16 , wherein a host computer using an application programming interface obtains a list of virtual machines on the host computer which are consuming storage of a particular service level objective.
18. The method of claim 14 , wherein the host computers send a vendor SCSI command to the data storage system to obtain the port congestion values.
19. The method of claim 18 , wherein the vendor SCSI command is either a first type that requests a specific port's congestion value and is repeated for each port, or a second type requesting congestion values for all the ports visible to the host computer, the vendor SCSI command being sent in a periodic manner and the returned data being updated into an internal data structure of the multipathing logic.
20. The method of claim 14 , wherein the data storage system includes front end emulations through which the host computers connect to storage resources, the front end emulations employing executor threads to provide a major part of front end processing of the storage IO commands.