Adaptive rate throttling for cloud-based service requests
Some service providers (e.g., Microsoft's Office 365 services) throttle application or service requests an organization, a user, or device can request from a specific application or a suite of applications/services. This throttling can significantly impact performance for, not only the requesting user/device, but also for other tenants associated with that organization. To limit this imposed external throttling, an intelligent internal adaptive throttling or rate limiting may be implemented. In an embodiment, the system monitors throttling imposed by the service provider. Using this information, the system determines an optimal rate of calls that can made for that service to maximize throughput and performance. The system continuously monitors the throttling and in response dynamically modifies the internal target rate.
1 . A method for managing service request throttling from a third-party service, comprising:
obtaining an internal throttling rate target,
wherein the internal throttling rate target represent a maximum number of service calls allowed per time interval to a third-party service provider,
wherein the internal throttling rate target is derived from at least one of an information management policy, a definition within source code, or dynamically calculated;
continuously determining a number of service requests to allow in accordance with the internal throttling rate target, wherein the determination is based, at least, on a total number of service requests already made;
periodically:
accessing performance data associated with service requests initiated to the third-party service provider;
determining a current external throttling percent for a preceding period based on a number of service requests throttled by the third-party service provider during the preceding period and an amount of time one or more of the respective service requests were instructed to wait for;
determining that the current external throttling percent falls outside a threshold range of a throttling percent target; and
in response to determining that the current external throttling percent falls outside a threshold range of the throttling percent target:
calculating a rate correction factor using the throttling percent target and the current external throttling percent, and
adjusting the internal throttling rate target by multiplying the rate correction factor by the ibternal throttling rate target.
2 . The method of claim 1 , wherein the internal throttling rate target is dynamically adjusted based on machine learning analysis of historical performance data and Recovery Point Objective (RPO) and Recovery Time Objective (RTO) requirements.
3 . The method of claim 1 , wherein determining whether the current external throttling percent is within a threshold range comprises comparing the current external throttling percent with a percent threshold range.
4 . The method of claim 1 , wherein the service requests comprise one or more of backup, migration, or data recovery operations performed on cloud-based application data.
5 . The method of claim 1 , further comprising obtaining performance data from a remote database or a storage medium accessible by a service request manager.
6 . The method of claim 1 , wherein the internal throttling rate target is adjusted at intervals that are dynamically shortened based on deviation between external throttling percent and the internal throttling rate target.
7 . The method of claim 1 , wherein the service requests are associated with a multi-tenant cloud application environment.
8 . The method of claim 1 , further comprising storing calculated throttling correction factors as part of performance data for use in subsequent adaptive throttling operations.
9 . The method of claim 1 , further comprising resetting the throttling percent target when external throttling percent continuously falls outside a predefined threshold range.
10 . A system for managing service request throttling from a third-party service, the system configured to:
using one or more computer processors:
obtain an internal throttling rate target,
wherein the internal throttling rate target represent a maximum number of service calls allowed per time interval to a third-party service provider,
wherein the internal throttling rate target is derived from at least one of an information management policy, a definition within source code, or dynamically calculated;
continuously determine a number of service requests to allow in accordance with the internal throttling rate target, wherein the determination is based, at least, on a total number of service requests already made;
periodically:
access performance data associated with service requests initiated to the third-party service provider;
determine a current external throttling percent for a preceding period based on a number of service requests throttled by the third-party service provider during the preceding period and an amount of time one or more of the respective service requests were instructed to wait for;
determine that the current external throttling percent falls outside a threshold range of a throttling percent target; and
in response to the determination that the current external throttling percent falls outside a threshold range of a throttling percent target:
calculate a rate correction factor using the throttling percent target and the current external throttling percent, and
adjust the internal throttling rate target by multiplying the rate correction factor by the internal throttling rate target.
11 . The system of claim 10 , wherein the internal throttling rate target is dynamically adjusted based on machine learning analysis of historical performance data and Recovery Point Objective (RPO) and Recovery Time Objective (RTO) requirements.
12 . The system of claim 10 , wherein determining whether the current external throttling percent is within a threshold range comprises comparing the current external throttling percent with a percent threshold range.
13 . The system of claim 10 , wherein the service requests comprise one or more of backup, migration, or data recovery operations performed on cloud-based application data.
14 . The system of claim 10 , further comprising obtaining performance data from a remote database or a storage medium accessible by a service request manager.
15 . The system of claim 10 , wherein the internal throttling rate target is adjusted at intervals that are dynamically shortened based on deviation between external throttling percent and the internal throttling rate target.
16 . The system of claim 10 , wherein the service requests are associated with a multi-tenant cloud application environment.
17 . The system of claim 10 , further comprising storing calculated throttling correction factors as part of performance data for use in subsequent adaptive throttling operations.
18 . The system of claim 10 , further comprising resetting the throttling percent target when external throttling percent continuously falls outside a predefined threshold range.