Method and system for achieving fair command processing in storage systems that implement command-associated priority queuing
In certain, currently available data-storage systems, incoming commands from remote host computers are subject to several levels of command-queue-depth-fairness-related throttles to ensure that all host computers accessing the data-storage systems receive a reasonable fraction of data-storage-system command-processing bandwidth to avoid starvation of one or more host computers. Recently, certain host-computer-to-data-storage-system communication protocols have been enhanced to provide for association of priorities with commands. However, these new command-associated priorities may lead to starvation of priority levels and to a risk of deadlock due to priority-level starvation and priority inversion. In various embodiments of the present invention, at least one additional level of command-queue-depth-fairness-related throttling is introduced in order to avoid starvation of one or more priority levels, thereby eliminating or minimizing the risk of priority-level starvation and priority-related deadlock.
1. An array controller comprising:
ports associated with one or more first communications media that link the array controller to one or more host computers;
ports associated with one or more second communications media that link the array controller to one or more data-storage targets; and
array-controller logic that
receives commands with associated priority levels from host computers through the first communications media, wherein each command includes within the command a priority field storing an associated priority level and wherein the associated priority level corresponds to a priority level within a host computer associated with the command,
queues received commands according to the priorities associated with the commands to an internal work queue, from which they are subsequently processed for delivery to the data-storage targets,
throttles command reception from host computers in order to control an overall command-reception rate,
throttles command reception from individual host computers in order to provide fair access to all host computers accessing the array controller, and
throttles command reception from one or more of all host computers according to the associated priority levels in order to provide fair access to each priority level in each host computer of all host computers accessing the array controller.
2. The array controller of claim 1 wherein the array-controller throttles command reception by:
for each received command,
determining whether the received command can be processed;
when the received command can be processed, queuing the received command to the internal work queue; and
when the received command cannot be processed, returning a busy response to the host computer that sent the command to the array controller.
3. The array controller of claim 2 wherein the array-controller, in order to throttle command reception from host computers in order to control an overall command-reception rate, determines whether each received command can be processed by:
determining whether more than a threshold number of entries are currently queued to the work queue.
4. The array controller of claim 2 wherein the array-controller, in order to throttle command reception from individual host computers in order to provide fair access to all host computers accessing the array controller, determines whether each received command can be processed by:
determining whether more than a threshold number of entries representing commands received from a particular host computer that sent the received command are currently queued to the work queue.
5. The array controller of claim 4 wherein the array controller, in determining whether each received command can be processed, further considers a rate of busy-response return to the particular host computer, average rate of command reception from active host computers, and additional command-reception-rate and command-processing-bandwidth considerations.
6. The array controller of claim 2 wherein the array-controller, in order to throttle command reception from host computers according to associated priorities in order to provide fair access to all priority levels of all host computers accessing the array controller, determines whether each received command can be processed by:
determining whether more than a threshold number of entries representing commands received from a particular priority level of the particular host computer that sent the received command are currently queued to the work queue.
7. The array controller of claim 6 wherein the array controller, in determining whether each received command can be processed, further considers a rate of busy-response return to particular priority levels of the particular host computer, average rate of command reception from various priority levels of active host computers, and additional priority-level- related command-reception-rate and command-processing-bandwidth considerations.
8. The array controller of claim 1 wherein the array controller throttles command reception independently for each port associated with one or more first communications media.
9. A method for managing command-reception rates in an array controller that includes ports associated with one or more first communications media that link the array controller to one or more host computers, ports associated with one or more second communications media that link the array controller to one or more data-storage targets, and array-controller logic, the method comprising:
receiving commands with associated priority levels from host computers through the first communications media, wherein each command includes within the command a priority field storing an associated priority level and wherein the associated priority level corresponds to a priority level within a host computer associated with the command,
queuing received commands according to the priorities associated with the commands to an internal work queue, from which they are subsequently processed for delivery to the data-storage targets,
throttling command reception from host computers in order to control an overall command-reception rate,
throttling command reception from individual host computers in order to provide fair access to all host computers accessing the array controller, and
throttling command reception from one or more of all host computers according to the associated priority levels in order to provide fair access to each priority level in each host computer of all host computers accessing the array controller.
10. The method of claim 9 further including throttling command reception by:
for each received command,
determining whether the received command can be processed;
when the received command can be processed, queuing the received command to the internal work queue; and
when the received command cannot be processed, returning a busy response to the host computer that sent the command to the array controller.
11. The method of claim 10 wherein, in order to throttle command reception from host computers in order to control an overall command-reception rate, determining whether each received command can be processed by:
determining whether more than a threshold number of entries are currently queued to the work queue.
12. The method of claim 10 wherein, in order to throttle command reception from individual host computers in order to provide fair access to all host computers accessing the array controller, determining whether each received command can be processed by:
determining whether more than a threshold number of entries representing commands received from a particular host computer that sent the received command are currently queued to the work queue.
13. The method of claim 12 wherein, in determining whether each received command can be processed, further considering a rate of busy-response return to the particular host computer, average rate of command reception from active host computers, and additional command-reception-rate and command-processing-bandwidth considerations.
14. The method of claim 10 wherein, in order to throttle command reception from host computers according to associated priorities in order to provide fair access to all priority levels of all host computers accessing the array controller, determining whether each received command can be processed by:
determining whether more than a threshold number of entries representing commands received from a particular priority level of the particular host computer that sent the received command are currently queued to the work queue.
15. The method of claim 14 wherein, in determining whether each received command can be processed, further considering a rate of busy-response return to particular priority levels of the particular host computer, average rate of command reception from various priority levels of active host computers, and additional priority-level-related command-reception-rate and command-processing-bandwidth considerations.
16. The method of claim 9 further including throttling command reception independently for each port associated with one or more first communications media.
17. Processor instructions stored in a computer readable memory that implement the method of claim 9 .
18. Firmware instructions stored in a computer readable memory that implement the method of claim 9 .