Dynamic quality of service implementation based upon resource saturation
Techniques are provided for dynamically implementing quality of service policies for a distributed storage system based upon resources saturation. A quality of service policy is defined for throttling I/O operations received by a node of the distributed storage system based upon whether resources of the node have become saturated. The quality of service policy is dynamically implemented based upon ever changing resource utilization and saturation. Dynamically implementing the quality of service policy improves the ability to efficiently utilize resources of the node compared to conventional static polices that cannot adequately react to such changing considerations and resource utilization/saturation. With conventional static policies, an administrator manually defines a minimum amount of guaranteed resources and/or a maximum resource usage cap that could be set to values that result in inefficient operation and resource starvation. Dynamically implementing the quality of service policy results in more efficient operation and mitigates resource starvation.
1 . A method comprising:
maintaining a queue, for each priority band specified by a resource saturation based quality of service policy, used to queue I/O operations targeting volumes of a distributed storage system implementing the resource saturation based quality of service policy, where each priority band specifies a different level of service to provide to the volumes of the distributed storage system;
assigning weights to each queue based upon the priority bands, wherein a first weight is assigned to a first queue based upon the first queue being maintained for a first priority band, a second weight is assigned to a second queue based upon the second queue being maintained for a second priority band, and a third weight is assigned to a third queue based upon the third queue being maintained for a third priority band;
queuing an I/O operation into at least one of the first queue, the second queue, or the third queue as a queued I/O operation based upon resource saturation monitored using the resource saturation based quality of service policy; and
utilizing the weights to select the queued I/O operation to dequeue and reevaluate for execution utilizing current resource saturation monitored using the resource saturation based quality of service policy.
2 . The method of claim 1 , comprising:
for each priority band, determining a count of volumes assigned to a priority band; and
inputting the weights and the counts of volumes into a probability function to select the queued I/O operation.
3 . The method of claim 1 , comprising:
utilizing the resource saturation based quality of service policy to dynamically set a throughput ceiling limit for the volume to throttle and shape I/O operations.
4 . The method of claim 1 , comprising:
for each priority band, determining a count of volumes assigned to a priority band; and
inputting the weights and the counts of volumes into a random number generator to select the queued I/O operation.
5 . The method of claim 1 , comprising:
modifying the resource saturation based quality of service policy based upon I/O statistics of a storage operating system hosted by the distributed storage system.
6 . The method of claim 1 , comprising:
performing write throttling upon incoming I/O operations directed to a cache of the distributed storage system using the resource saturation based quality of service policy to protect distributed backend storage of the distributed storage system from being overloaded.
7 . The method of claim 1 , comprising:
configuring a quality of service priority scheduler to schedule I/O operations directed to a first volume according to a first proportion corresponding to the first priority band and schedule I/O operations directed to a second volume according to a second proportion based upon the second volume corresponding to the second priority band.
8 . The method of claim 7 , comprising:
configuring the quality of service priority scheduler to schedule I/O operations directed to a third volume according to a third proportion based upon the third volume corresponding to the third priority band.
9 . A non-transitory machine readable medium comprising instructions, which when executed by a machine, causes the machine to:
maintain a queue, for each priority band specified by a resource saturation based quality of service policy, used to queue I/O operations targeting volumes of a distributed storage system implementing the resource saturation based quality of service policy, where each priority band specifies a different level of service to provide to the volumes of the distributed storage system;
assign eights to each queue based upon the priority bands, wherein a first weight is assigned to a first queue based upon the first queue being maintained for a first priority band, a second weight is assigned to a second queue based upon the second queue being maintained for a second priority band, and a third weight is assigned to a third queue based upon the third queue being maintained for a third priority band;
queue an I/O operation into at least one of the first queue, the second queue, or the third queue as a queued I/O operation based upon resource saturation monitored using the resource saturation based quality of service policy; and
utilize the weights to select the queued I/O operation to dequeue and reevaluate for execution utilizing current resource saturation monitored using the resource saturation based quality of service policy.
10 . The non-transitory machine readable medium of claim 9 , wherein the instructions further cause the machine to:
determine whether a credit pool comprises available credits to allocate to the I/O operation for execution, wherein credits within the credit pool are allocated based upon resource utilization.
11 . The non-transitory machine readable medium of claim 9 , wherein the instructions further cause the machine to:
in response to utilization of a resource exceeding a threshold, routing the I/O operation to a file system of the distributed storage system for execution.
12 . The non-transitory machine readable medium of claim 9 , wherein the instructions further cause the machine to:
utilize an I/O processing rate recommendation for I/O throttling of I/O operations to either route the I/O operations to a file system of the distributed storage system for execution or queue the I/O operations.
13 . The non-transitory machine readable medium of claim 9 , wherein the instructions further cause the machine to:
enforce, as part of processing the queue, a node resource saturation point based upon processor load of the distributed storage system.
14 . The non-transitory machine readable medium of claim 9 , wherein the instructions further cause the machine to:
determine whether to queue the I/O operations based upon round trip operation completion statistics.
15 . A computing device, of a distributed storage system, comprising:
a memory comprising machine executable code; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the computing device to:
maintain a queue, for each priority band specified by a resource saturation based quality of service policy, used to queue I/O operations targeting volumes of a distributed storage system implementing the resource saturation based quality of service policy, where each priority band specifies a different level of service to provide to the volumes of the distributed storage system;
assign weights to each queue based upon the priority bands, wherein a first weight is assigned to a first queue based upon the first queue being maintained for a first priority band, a second weight is assigned to a second queue based upon the second queue being maintained for a second priority band, and a third weight is assigned to a third queue based upon the third queue being maintained for a third priority band;
queue an I/O operation into the first queue or the second queue based upon resource saturation monitored using the resource saturation based quality of service policy; and
utilize the weights to select the queued I/O operation to dequeue and reevaluate for execution utilizing current resource saturation monitored using the resource saturation based quality of service policy.
16 . The computing device of claim 15 , wherein the machine executable code causes the computing device to:
dispatch a queued I/O operation to a file system of the distributed storage system or aborts the queued I/O operation based upon the queued I/O operation pending in the first queue or the second queue for greater than a threshold timespan.
17 . The computing device of claim 15 , wherein the machine executable code causes the computing device to:
modify the resource saturation based quality of service policy based upon I/O statistics of a storage operating system hosted by the distributed storage system.
18 . The computing device of claim 15 , wherein the machine executable code causes the computing device to:
perform write throttling upon incoming I/O operations directed to a cache of the distributed storage system using the resource saturation based quality of service policy to protect distributed backend storage of the distributed storage system from being overloaded.
19 . The computing device of claim 15 , wherein the machine executable code causes the computing device to:
configure a quality of service priority scheduler to schedule I/O operations directed to a first volume according to a first proportion corresponding to the first priority band and schedule I/O operations directed to a second volume according to a second proportion based upon the second volume corresponding to the second priority band.
20 . The computing device of claim 19 , wherein the machine executable code causes the computing device to:
configure the quality of service priority scheduler to schedule I/O operations directed to a third volume according to a third proportion based upon the third volume corresponding to the third priority band.