DYNAMIC RATE LIMITER TO CONTROL SERVER ACCESS
Systems, methods, and apparatus related to server access. In one approach, a computing device accesses one or more external servers through a gateway. A rate at which the computing device accesses the servers is controlled based on monitoring the context of a computing environment. Examples of the context include available spare processing bandwidth and/or processing rates for a given user or tenant. The rate is regulated by a dynamic rate limiter and is updated in real-time to reflect any changes to the monitored context.
1 . A system comprising:
a gateway configured to provide network access to at least one server; and
at least one computing device configured to:
execute at least one service that accesses the server via the gateway;
control at least one rate at which the service accesses the server;
monitor a context associated with the service; and
dynamically update the rate at which the service accesses the server to reflect any changes to the monitored context.
2 . The system of claim 1 , wherein the service accesses the server using at least one network, and monitoring the context comprises receiving data regarding activity associated with network communication.
3 . The system of claim 1 , wherein:
the service is an application service that uses at least one input from at least one dependent service, or provides at least one output to the dependent service; and
monitoring the context comprises receiving data regarding the dependent service.
4 . The system of claim 1 , wherein the service receives communications comprising at least one of:
a communication regarding an event;
a message; or
a request.
5 . The system of claim 4 , wherein the service is an application service that receives the communications for processing.
6 . The system of claim 4 , further comprising a message broker that queues messages for processing by the application service, wherein processing the messages includes making calls to the server via the gateway.
7 . The system of claim 1 , further comprising a rate limiter configured to provide a respective token to each of a plurality of first services, wherein each token corresponds to a number of calls that the respective first service is permitted to make to the server via the gateway.
8 . The system of claim 7 , wherein the rate limiter is further configured to control a rate of providing tokens to the first services.
9 . The system of claim 8 , wherein:
updating the rate at which the service accesses the server comprises determining new rates based on the monitored context;
the rate limiter is further configured to instantiate multiple new instances of a static rate limiter per tenant, per user, or per account; and
each of the new instances is configured using one of the new rates.
10 . A method comprising:
accessing at least one server by a plurality of application services, wherein the application services are executing on at least one computing device and access the server through at least one gateway;
controlling at least one rate at which the application services access the server;
receiving monitoring data regarding at least one network or computing device;
evaluating the monitoring data; and
dynamically updating the at least one rate at which the application services access the server based on evaluating the monitoring data.
11 . The method of claim 10 , wherein evaluating the monitoring data comprises providing a result from a machine learning model that uses the received monitoring data as an input.
12 . The method of claim 10 , wherein the monitoring data includes data indicating an error associated with a service that is upstream or downstream to one or more of the application services.
13 . The method of claim 10 , further comprising:
determining that first events received by a first application service exceed an arrival rate threshold; and
in response to determining that the first events exceed the arrival rate threshold, moving the first events to an overflow queue.
14 . The method of claim 13 , further comprising:
identifying spare bandwidth in computing resources used to provide services to a first tenant or user; and
in response to identifying the spare bandwidth, adjusting a rate limit for processing the first events in the overflow queue.
15 . The method of claim 14 , wherein identifying the spare bandwidth comprises analyzing a current event arrival rate for the first tenant or user, and an event processing rate for the first tenant or user.
16 . A system comprising:
an event generator configured to gather data regarding user activity and generate messages based on the gathered data;
memory configured to store the messages in a queue;
a message broker configured to retrieve the messages from the queue for processing by at least one first service; and
at least one processing device configured to execute a rate limiter to:
monitor characteristics of computing resources; and
control, based on the monitored characteristics, at least one rate limit for the first service.
17 . The system of claim 16 , further comprising a data store configured to store a respective rate limit configuration for each of a plurality of tenants or users.
18 . The system of claim 16 , wherein the rate limit defines a maximum number of application programming interface (API) calls to a server that can be made by the first service.
19 . The system of claim 16 , wherein monitoring the characteristics includes analyzing event arrival or message processing rates for each of a plurality of tenants or users.
20 . The system of claim 16 , wherein monitoring the characteristics further includes monitoring operating or performance metrics for the event generator.