Weighted resource cost matrix scheduler
A scheduler for a storage node uses multi-dimensional weighted resource cost matrices to schedule processing of IOs. A separate matrix is created for each computing node of the storage node via machine learning or regression analysis. Each matrix includes distinct dimensions for each emulation of the computing node for which the matrix is created. Each dimension includes modeled costs in terms of amounts of resources of various types required to process an IO of various IO types. An IO received from a host by a computing node is not scheduled for processing by that computing node unless enough resources are available at each emulation of that computing node. If enough resources are unavailable at an emulation, then the IO is forwarded to a different computing node that has enough resources at each of its emulations. A weighted resource cost for processing the IO is calculated and used to determine scheduling priority. The weights or regression coefficients from the model may be used to calculate weighted resource cost.
1. An apparatus comprising:
a plurality of fabric-interconnected computing nodes that receive IO (Input-Output operation) commands from a plurality of hosts, each computing node comprising a host adapter, a drive adapter, a remote adapter, and an operating system;
a plurality of managed storage drives; and
a scheduler that:
creates a multi-dimensional weighted resource cost matrix for each of the computing nodes, each of the cost matrices comprising a host adapter dimension, a drive adapter dimension, a remote adapter dimension, and an operating system dimension, each dimension comprising an indication of numbers of resources for each of a plurality of resource types required for each of a plurality of IO types; and
identifies, for each emulation of a first computing node of the plurality of fabric-interconnected computing nodes, a resource cost to process a first IO received by the first computing node from one of the hosts, the scheduler using the multi-dimensional weighted resource cost matrices to calculate free resources available at each emulation of the first computing node and scheduling processing of the first IO responsive to determining that the available free resources at each emulation of the first computing node exceed the identified resource cost for each corresponding emulation of the first computing node.
2. The apparatus of claim 1 wherein the multi-dimensional weighted resource cost matrices comprise resource type weighting coefficients, and wherein the scheduler calculates a weighted resource cost for processing the first IO and determines an order in which the first IO will be processed by the first computing node relative to other scheduled IOs based on the weighted resource cost.
3. A method comprising:
in a data storage system comprising a plurality of computing nodes that receive IO (Input-Output operation) commands from a plurality of hosts, each computing node comprising a plurality of different emulations, scheduling an IO received by a first computing node by:
creating a multi-dimensional weighted resource cost matrix for each of the computing nodes, each of the cost matrices comprising a host adapter dimension, a drive adapter dimension, a remote adapter dimension, and an operating system dimension, each dimension comprising an indication of numbers of resources for each of a plurality of resource types required for each of a plurality of IO types;
identifying a resource cost to process the received IO for each emulation of the first computing node;
calculating free resources available at each emulation of the first computing node using the multi-dimensional weighted resource cost matrices; and
scheduling processing of the received IO responsive to determining that the available free resources at each emulation of the first computing node exceed the identified resource cost for each corresponding emulation of the first computing node.
4. The method of claim 3 wherein identifying the resource cost to process the received IO for each emulation of the first computing node comprises identifying one or more of queue depth, processor utilization, local memory utilization, shared memory utilization, bandwidth utilization of Fibre Channel hardware, iscsi hardware, nvme hardware, bandwidth utilization of internal InfiniBand hardware across different boards, compression IOs queue depth and compression hardware bandwidth, deduplication IOs queue depth and deduplication hardware bandwidth, encryption hardware bandwidth and queue depth and memory bus bandwidth.
5. The method of claim 3 wherein identifying the resource cost to process the received IO for each emulation of the first computing node comprises identifying the resource cost for each of a host adapter, a drive adapter, a remote replication adapter, a local replication and data services adapter, and an operating system.
6. The method of claim 3 comprising calculating a weighted resource cost for processing the received IO.
7. The method of claim 6 comprising determining an order in which the received IO will be processed by the first computing node relative to other scheduled IOs based on the weighted resource cost.
8. The method of claim 3 comprising determining that available free resources at each emulation of a second computing node exceed the identified resource cost for each corresponding emulation of the second computing node responsive to determining that at least one of the available free resources at one or more emulation of the first computing node does not exceed the identified resource cost for each corresponding emulation of the first computing node.
9. The method of claim 8 comprising determining that available free resources at each emulation of a third computing node exceed the identified resource cost for each corresponding emulation of the third computing node responsive to determining that at least one of the available free resources at one or more emulation of the first computing node does not exceed the identified resource cost for each corresponding emulation of the first computing node, and further comprising selecting the second computing node responsive to the second computing node having greater available free resources than the third computing node.
10. The method of claim 9 comprising calculating a weighted resource cost for processing the received IO.
11. The method of claim 10 comprising determining an order in which the received IO will be processed by the second computing node relative to other scheduled IOs based on the weighted resource cost.
12. The method of claim 3 comprising adjusting resource allocation for an emulation of the first computing node based on recency, frequency, and response time.
13. An apparatus comprising:
a plurality of fabric-interconnected computing nodes that receive IO (Input-Output operation) commands from a plurality of hosts, each computing node comprising a plurality of different emulations;
a plurality of managed storage drives; and
a scheduler that creates a multi-dimensional weighted resource cost matrix for each of the computing nodes, each of the cost matrices comprising a host adapter dimension, a drive adapter dimension, a remote adapter dimension, and an operating system dimension, each dimension comprising an indication of numbers of resources for each of a plurality of resource types required for each of a plurality of IO types, and identifies, for each emulation of a first computing node of the plurality of fabric-interconnected computing nodes, a resource cost to process a first IO received by the first computing node from one of the hosts, the scheduler using the multi-dimensional weighted resource cost matrices to calculate free resources available at each emulation of the first computing node and scheduling processing of the first IO responsive to determining that the available free resources at each emulation of the first computing node exceed the identified resource cost for each corresponding emulation of the first computing node.
14. The apparatus of claim 13 wherein the matrices comprise one or more resource types selected from the group consisting of: IO queue depth, processor utilization, local memory utilization, shared memory utilization, bandwidth utilization of Fibre Channel hardware, iscsi hardware, nvme hardware, bandwidth utilization of internal InfiniBand hardware across different boards, compression IOs queue depth and compression hardware bandwidth, deduplication IOs queue depth and deduplication hardware bandwidth, encryption hardware bandwidth and queue depth and memory bus bandwidth.
15. The apparatus of claim 13 wherein the emulations of the first computing node comprise a host adapter, a drive adapter, a remote replication adapter, a local replication and data service adapter, and an operating system.
16. The apparatus of claim 13 wherein the scheduler calculates a weighted resource cost for processing the first IO.
17. The apparatus of claim 16 wherein the scheduler determines an order in which the first IO will be processed by the first computing node relative to other scheduled IOs based on the weighted resource cost.
18. The apparatus of claim 13 wherein the scheduler determines whether other computing nodes have enough available free resources at each emulation to process the first IO in response to determining that at least one emulation of the first computing node lacks enough available resources to process the first IO.
19. The apparatus of claim 18 wherein the scheduler selects one of the other computing nodes having enough available free resources at each emulation to process the first IO based on a comparison of amounts of available free resources.
20. The apparatus of claim 13 wherein the scheduler adjusts resource allocation for an emulation of the first computing node based on recency, frequency, and response time.