ASYNCHRONOUS STATISTIC-BASED RATE LIMITING IN DISTRIBUTED SYSTEM
In an example embodiment, rate limiting is performed at the instance level (i.e., locally), but utilizing throughput statistics of other instances. These statistics may be measured locally by each instance and then transmitted to a central store, where they are aggregated. Each instance is then able to asynchronously request the aggregated statistics from the central store and use this information to manage the parameters of its own local rate limiter.
1 . A method, comprising:
obtaining, by a computing system from a data source, a training dataset comprising time of day information and node type information, wherein the node type information comprises service and machine information for a given node represented in the training dataset;
pre-processing, by the computing system, the training dataset;
training, by the computing system, a machine learning model to predict modifications of capacity or fill rate based on a node type of a node that is to execute a service client instance and corresponding time data;
providing, by the computing system, the machine learning model to the service client instance of the node;
generating, by the service client instance of the node, an asynchronous call to a central store for a capacity or a fill rate for a service provided via the service client instance;
executing, by the service client instance of the node, the machine learning model using a time of day and a node type of the node to generate a predicted modification of the capacity or the fill rate; and
modifying, by the service client instance of the node, a rate limit for requests for the service based on the predicted modification of the capacity or the fill rate.
2 . The method of claim 1 , wherein the training dataset further comprises throughput information.
3 . The method of claim 1 , further comprising:
applying, by the computing system, a MapReduce function to the training dataset.
4 . The method of claim 1 , wherein the machine learning model comprises one or more of a neural network, a Bayesian network, or a logistic regression model.
5 . The method of claim 1 , further comprising:
establishing, by the service client instance of the node, the rate limit for requests for the service based on the capacity of the service and a total number of client instances connected to the service.
6 . The method of claim 1 , further comprising:
adjusting, by the service client instance of the node, the rate limit based on a number of requests for the service and a total capacity of the service.
7 . The method of claim 6 , further comprising:
reporting, by the service client instance of the node, the number of requests for the service received at the service client instance to the central store.
8 . The method of claim 1 , further comprising:
enforcing, by the service client instance of the node, the rate limit for requests for the service.
9 . The method of claim 8 , wherein enforcing the rate limit comprises:
determining, by the service client instance of the node, that a received request causes the rate limit to be exceeded; and
delaying, by the service client instance of the node, transmission of the received request upon determining that the received request causes the rate limit to be exceeded.
10 . The method of claim 1 , further comprising:
determining, by the service client instance of the node, that a first time period associated with the rate limit has expired; and
generating, by the service client instance of the node, a second rate limit for a second time period upon determining that the first time period has expired.
11 . A system, comprising:
a computing system configured to:
obtain, from a data source, a training dataset comprising time of day information and node type information, wherein the node type information comprises service and machine information for a given node represented in the training dataset;
pre-process the training dataset;
train a machine learning model to predict modifications of capacity or fill rate based on a node type of a node that is to execute a service client instance and corresponding time data;
provide the machine learning model to the service client instance of the node; and
a service client instance of the node configured to:
generate an asynchronous call to a central store for a capacity or a fill rate for a service provided via the service client instance;
execute the machine learning model using a time of day and a node type of the node to generate a predicted modification of the capacity or the fill rate; and
modify a rate limit for requests for the service based on the predicted modification of the capacity or the fill rate.
12 . The system of claim 11 , wherein the training dataset further comprises throughput information.
13 . The system of claim 11 , wherein the computing system is further configured to:
apply a MapReduce function to the training dataset.
14 . The system of claim 11 , wherein the machine learning model comprises one or more of a neural network, a Bayesian network, or a logistic regression model.
15 . The system of claim 11 , wherein the service client instance of the node is further configured to:
establish the rate limit for requests for the service based on the capacity of the service and a total number of client instances connected to the service.
16 . The system of claim 11 , wherein the service client instance of the node is further configured to:
adjust the rate limit based on a number of requests for the service and a total capacity of the service.
17 . The system of claim 16 , wherein the service client instance of the node is further configured to:
report the number of requests for the service received at the service client instance to the central store.
18 . The system of claim 11 , wherein the service client instance of the node is further configured to:
enforce the rate limit for requests for the service.
19 . The system of claim 18 , wherein the service client instance of the node is further configured to:
determine that a received request causes the rate limit to be exceeded; and
delay transmission of the received request upon determining that the received request causes the rate limit to be exceeded.
20 . The system of claim 11 , wherein the service client instance of the node is further configured to:
determine that a first time period associated with the rate limit has expired; and
generate a second rate limit for a second time period upon determining that the first time period has expired.