System and method for throttling service requests having non-uniform workloads
A system that provides services to clients may receive and service requests, various ones of which may require different amounts of work. The system may determine whether it is operating in an overloaded or underloaded state based on a current work throughput rate, a target work throughput rate, a maximum request rate, or an actual request rate, and may dynamically adjust the maximum request rate in response. For example, if the maximum request rate is being exceeded, the maximum request rate may be raised or lowered, dependent on the current work throughput rate. If the target or committed work throughput rate is being exceeded, but the maximum request rate is not being exceeded, a lower maximum request rate may be proposed. Adjustments to the maximum request rate may be made using multiple incremental adjustments. Service request tokens may be added to a leaky token bucket at the maximum request rate.
1 . A method, comprising:
performing by a Web service:
monitoring performance of a plurality of service requests of a particular one or more clients received at the Web service according to a requested target rate of units of work to service requests associated with the particular one or more clients at the Web service and within a maximum allowable request rate for the particular one or more clients;
wherein the monitoring comprises:
measuring respective numbers of units of work performed to service a portion of the plurality of service requests at the Web service; and
comparing, at the Web service, an observed rate of the respective numbers of units of work performed to service the portion of the plurality of service requests with respect to the requested target rate of the units of work; and
based, at least in part, on the comparison and a configurable value that determines respective amounts of one or more incremental adjustments of the maximum allowable request rate for the particular one or more clients in one or more adjustment cycles, allocating one or more resources to service additional service requests received at the network-based service according to a plurality of incremental adjustments to the maximum allowable request rate used to manage performance of the additional service requests received at the Web service subsequent to the plurality of incremental adjustments to the maximum allowable request rate.
2 . The method of claim 1 ,
wherein the observed rate exceeds the requested target rate; and
wherein the plurality of incremental adjustments comprises:
determining the amount by which to lower the maximum allowable request rate; and
lowering the maximum allowable request rate by the determined amount.
3 . The method of claim 2 ,
wherein the maximum allowable request rate represents the maximum number of requests that can be accepted for servicing per second.
4 . The method of claim 1 , wherein the observed rate is an average of rates measured in two or more sample periods.
5 . The method of claim 1 ,
wherein the plurality of incremental adjustments comprises:
determining an amount by which to raise the maximum allowable request rate; and
raising the maximum allowable request rate by the determined amount.
6 . A system, comprising:
one or more processors; and
a memory coupled to the one or more processors and storing program instructions that when executed by the one or more processors cause the one or more processors to implement a Web service, configured to:
monitor performance of a plurality of service requests of a particular one or more clients received at the Web service according to a requested target rate of units of work to service requests at the Web service associated with the particular one or more clients and within a maximum request rate for the particular one or more clients;
wherein to monitor performance, the Web service is configured to:
measure respective numbers of units of work performed to service a portion of the plurality of service requests at the Web service;
compare, at the Web service, an observed rate of the respective numbers of units of work performed to service the portion of the plurality of service requests with respect to the requested target rate of units of work; and
based, at least in part, on the comparison and a configurable value that determines respective amounts of a plurality of incremental adjustments of the maximum allowable request rate for the particular one or more clients in a plurality of adjustment cycles, allocate one or more resources to service additional service requests received at the network-based service according to a plurality of incremental adjustments to the maximum allowable request rate used to manage performance of the additional service requests received at the Web service subsequent to the plurality of incremental adjustments to the maximum allowable request rate.
7 . The system of claim 6 ,
wherein the observed rate at which work was performed exceeds the requested target rate for performing work when servicing requests; and
wherein the plurality of incremental adjustments:
determine an amount by which to lower the maximum allowable request rate; and
lower the maximum allowable request rate by the determined amount.
8 . The system of claim 7 ,
wherein the maximum allowable request rate represents the maximum number of requests that can be accepted for servicing per second; and
wherein the requested target rate of units of work is specified in terms of a number of units of work performed per second.
9 . The system of claim 7 , wherein the requested target rate of units of work comprises a committed rate for performing work that is specified by the client-specific policy.
10 . The system of claim 6 , wherein the observed rate of the respective numbers of units of work represents an average of the rates at which work was performed in two or more sample periods.
11 . The system of claim 10 , wherein the the average represents a moving average over a particular number of sample periods.
12 . The system of claim 6 ,
wherein the observed rate at which work was performed is less than the requested target rate for performing work when servicing one or more requests; and
wherein the plurality of incremental adjustments:
determine an amount by which to raise the maximum allowable request rate; and
raise the maximum allowable request rate by the determined amount.
13 . The system of claim 6 , wherein the plurality of incremental adjustments are performed according to an adjustment cycle.
14 . One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices, cause the one or more computing devices to implement a Web service that implements:
monitoring performance of a plurality of service requests associated with a particular one or more clients received at the Web service according to a requested target rate of units of work to service requests associated the particular one or more clients at the Web service and within a maximum request rate for the particular one or more clients;
wherein, in monitoring, the program instructions cause the one or more computing devices to implement:
measuring respective numbers of units of work performed to service a portion of the plurality of service requests at the Web service;
comparing, at the Web service, an observed rate of the respective numbers of units of work performed to service the portion of the plurality of service requests with respect to the requested target rate of units of work; and
based, at least in part, on the comparison and a configurable value that determines respective amounts of a plurality of incremental adjustments of the maximum allowable request rate for the particular one or more clients in a plurality of adjustment cycles, allocating one or more resources to service additional service requests received at the network-based service according to a plurality of incremental adjustments to the maximum allowable request rate used to manage performance of the additional service requests received at the Web service subsequent to the plurality of incremental adjustments to the maximum allowable request rate.
15 . The one or more non-transitory, computer-readable storage media of claim 14 ,
wherein the observed rate at which work was performed exceeds the requested target rate for performing work when servicing requests; and
wherein the plurality of incremental adjustments, comprises:
determining an amount by which to lower the maximum allowable request rate; and
lowering the maximum allowable request rate by the determined amount.
16 . The one or more non-transitory, computer-readable storage media of claim 15 ,
wherein the maximum allowable request rate represents the maximum number of requests that can be accepted for servicing per second; and
wherein the requested target rate of units of work is specified in terms of a number of units of work performed per second.
17 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein the requested target rate of units of work comprises a committed rate for performing work that is specified by a client-specific policy.
18 . The one or more non-transitory, computer-readable storage media of claim 14 , wherein the observed rate of the respective numbers of units of work performed represents an average of the rates at which work was performed in two or more sample periods.
19 . The one or more non-transitory, computer-readable storage media of claim 18 , wherein the average represents a moving average over a particular number of sample periods.
20 . The one or more non-transitory, computer-readable storage media of claim 14 ,
wherein the observed rate at which work was performed is less than the requested target rate for performing work when servicing one or more requests; and
wherein the plurality of incremental adjustments, comprises:
determining an amount by which to raise the maximum allowable request rate; and
raising the maximum allowable request rate by the determined amount.