Asynchronous statistic-based rate limiting in distributed system
In an example embodiment, rate limiting is performed at the instance level (i.e., locally), but utilizing throughput statistics of other instances. These statistics may be measured locally by each instance and then transmitted to a central store, where they are aggregated. Each instance is then able to asynchronously request the aggregated statistics from the central store and use this information to manage the parameters of its own local rate limiter.
1. A method comprising:
at a client instance of a service:
establishing a rate limit for requests for the service received at the client instance;
tracking a number of requests for the service received at the client instance during a first time period, while enforcing the rate limit;
reporting the number of requests for the service received at the client instance during the first time period to a central store, wherein the central store is configured to receive and store reports from a plurality of client instances of the service connected to the service during the first time period;
making an asynchronous call to the central store for aggregated information regarding a total number of the plurality of client instances of the service connected to the service during the first time period and the number of requests for the service received at all of the plurality of client instances of the service during the first time period;
adjusting the rate limit based on the total number of the plurality of client instances of the service connected to the service during the first time period and the number of requests for the service received at all of the plurality of client instances of the service during the first time period; and
modifying the adjusted rate limit based on a prediction from a machine learned model,
wherein the machine learned model is trained using time of day information to predict a modification based on node type of a node running the service client instance, wherein the node type is a combination of the service and a type of machine running the node.
2. The method of claim 1 , wherein the establishing the rate limit includes dividing a capacity of the service by the total number of the plurality of client instances connected to the service.
3. The method of claim 1 , wherein the adjusting the rate limit includes subtracting a number of requests for the service received at all of the plurality of client instances of the service during the time period from a total capacity of the service, dividing the difference by the total number of the plurality of client instances of the service connected to the service during the first time period, and adding a result of the dividing to the number of requests for the service received at the client instance during the first time period.
4. The method of claim 1 , further comprising repeating the tracking, reporting, making, and adjusting for a second time period equal in length to the first time period.
5. The method of claim 1 , wherein the machine learned model is trained using time of day information to predict a modification based on current time of day.
6. A system comprising:
one or more processors; and
a memory storing instructions that, when executed by at least one processor among the one or more processors, cause the at least one processor execute a client instance of a service to perform operations comprising:
establishing a rate limit for requests for the service received at the client instance;
tracking a number of requests for the service received at the client instance during a first time period, while enforcing the rate limit;
reporting the number of requests for the service received at the client instance during the first time period to a central store, wherein the central store is configured to receive and store reports from a plurality of client instances of the service connected to the service during the first time period;
making an asynchronous call to the central store for aggregated information regarding a total number of the plurality of client instances of the service connected to the service during the first time period and the number of requests for the service received at all of the plurality of client instances of the service during the first time period;
adjusting the rate limit based on the total number of the plurality of client instances of the service connected to the service during the first time period and the number of requests for the service received at all of the plurality of client instances of the service during the first time period; and
modifying the adjusted rate limit based on a prediction from a machine learned model,
wherein the machine learned model is trained using time of day information to predict a modification based on node type of a node running the service client instance, wherein the node type is a combination of the service and a type of machine running the node.
7. The system of claim 6 , wherein the establishing the rate limit includes dividing a capacity of the service by the total number of the plurality of client instances connected to the service.
8. The system of claim 6 , wherein the adjusting the rate limit includes subtracting a number of requests for the service received at all of the plurality of client instances of the service during the time period from a total capacity of the service, dividing the difference by the total number of the plurality of client instances of the service connected to the service during the first time period, and adding a result of the dividing to the number of requests for the service received at the client instance during the first time period.
9. The system of claim 6 , wherein the instructions further comprise repeating the tracking, reporting, making, and adjusting for a second time period equal in length to the first time period.
10. The system of claim 6 , wherein the machine learned model is trained using time of day information to predict a modification based on current time of day.
11. A non-transitory machine-readable medium comprising instructions which, when read by a machine, cause the machine to perform operations comprising:
at a client instance of a service:
establishing a rate limit for requests for the service received at the client instance;
tracking a number of requests for the service received at the client instance during a first time period, while enforcing the rate limit;
reporting the number of requests for the service received at the client instance during the first time period to a central store, wherein the central store is configured to receive and store reports from a plurality of client instances of the service connected to the service during the first time period;
making an asynchronous call to the central store for aggregated information regarding a total number of the plurality of client instances of the service connected to the service during the first time period and the number of requests for the service received at all of the plurality of client instances of the service during the first time period;
adjusting the rate limit based on the total number of the plurality of client instances of the service connected to the service during the first time period and the number of requests for the service received at all of the plurality of client instances of the service during the first time period; and
modifying the adjusted rate limit based on a prediction from a machine learned model,
wherein the machine learned model is trained using time of day information to predict a modification based on node type of a node running the service client instance, wherein the node type is a combination of the service and a type of machine running the node.
12. The non-transitory machine-readable medium of claim 11 , wherein the establishing the rate limit includes dividing a capacity of the service by the total number of the plurality of client instances connected to the service.
13. The non-transitory machine-readable medium of claim 11 , wherein the adjusting the rate limit includes subtracting a number of requests for the service received at all of the plurality of client instances of the service during the time period from a total capacity of the service, dividing the difference by the total number of the plurality of client instances of the service connected to the service during the first time period, and adding a result of the dividing to the number of requests for the service received at the client instance during the first time period.
14. The non-transitory machine-readable medium of claim 11 , wherein the instructions further comprise repeating the tracking, reporting, making, and adjusting for a second time period equal in length to the first time period.
15. The non-transitory machine-readable medium of claim 11 , wherein the machine learned model is trained using time of day information to predict a modification based on current time of day.