Fair queuing of request tasks spawned by requests to execute generative operations
Fair queuing of request tasks spawned by requests to execute generative operations such as, for example, graph query language requests to execute a graph query language query, mutation, or subscription operations. Queuing techniques are used to prevent a heavy generative operation from dominating usage of computing resources of a host that executes many generative operations concurrently including a mix of heavy and normal generative operations. Generative operations are analyzed and classified as heavy or normal as the request tasks they spawn are being executed. If a generative operation is classified as heavy, then subsequent request tasks spawned by the heavy generative operation are added to an overload queue while request tasks spawned by concurrently executing normal generative operations as added to a main queue. For fairness, request tasks are polled from the main queue for execution at greater frequency than request tasks in the overload queue.
1. A method comprising:
receiving a graph query language request specifying a graph query language operation to be executed;
executing the graph query language operation over a time period to yield an operation result; and
sending the operation result;
during the time period, dequeuing, from a main queue, and executing each request task of a first set of request tasks spawned by executing the graph query language operation;
during the time period, determining a request cost index reflecting an amount of computing resources used to execute the first set of request tasks;
during the time period, determining a cardinality of the first set of request tasks spawned by executing the graph query language operation;
during the time period, determining, based on the request cost index or the cardinality, to queue, in an overload queue, subsequent request tasks spawned by executing the graph query language operation;
during the time period, queuing, in the overload queue, a second set of request tasks spawned by executing the graph query language operation; and
during the time period, dequeuing, from the second queue, and executing each request task of the second set of request tasks.
2. The method of claim 1 , further comprising:
during the time period, sending a request task spawned by executing the graph query language operation directly to a thread pool, bypassing the main queue and the overload queue, based on determining there is an idle thread in the thread pool.
3. The method of claim 1 , further comprising:
classifying the graph query language operation as normal or heavy based on the request cost index or the cardinality exceeding a respective threshold.
4. A method comprising:
receiving a request to execute a generative operation;
executing the generative operation over a time period to yield an operation result;
sending the operation result;
during the time period, dequeuing, from a first queue, and executing each request task of a first set of request tasks spawned by executing the generative operation;
during the time period, determining a request cost index reflecting an amount of computing resources used to execute the first set of request tasks;
during the time period, determining, based on the request cost index, to queue, in a second queue, subsequent request tasks spawned by executing the generative operation;
during the time period, queuing, in the second queue, a second set of request tasks spawned by executing the generative operation; and
during the time period, dequeuing, from the second queue, and executing each request task of the second set of request tasks.
5. The method of claim 4 , further comprising:
during the time period, sending a request task spawned by executing the generative operation directly to a thread pool, bypassing the first queue and the second queue, based on determining there is an idle thread in the thread pool.
6. The method of claim 4 , further comprising:
classifying, during the period of time, the generative operation as normal or heavy based on the request cost index exceeding a threshold.
7. The method of claim 4 , wherein:
the first set of request tasks are dequeued, from the first queue, and executed at a first rate;
the second set of request tasks are dequeued, from the second queue, and executed at a second rate; and
the first rate is higher than the second rate.
8. The method of claim 4 , further comprising:
determining, during the period of time, a set of node request cost indexes for the first set of request tasks; and
determining, during the period of time, the request cost index based on a sum of the set of node request cost indexes.
9. The method of claim 4 , further comprising:
determining an average request cost index of a plurality of generative operations requested by a same user or customer as a user or customer that requested the generative operation;
during the period of time, initially classifying the generative operation as normal based on determining that the average request cost index is below a threshold; and
during the period of time, queueing the first set of request tasks in the first queue based on initially classifying the generative operation as normal.
10. The method of claim 4 , further comprising:
rejecting, during the period of time, a request task for execution spawned by executing the generative operation based on determining that the second queue is full; and
executing, during the period of time, the rejected request task in a calling thread.
11. The method of claim 4 , where the computing resources reflected by the request cost index comprise processor and memory resources of a host that executes the generative operation.
12. The method of claim 4 , wherein a host concurrently executes the generative operation with a plurality of other generative operations.
13. The method of claim 4 , further comprising:
determining, during the period of time, an amount of memory of a host allocated for the first set of request tasks; and
determining, during the period of time, the request cost index based on the amount of memory allocated.
14. The method of claim 4 , further comprising:
determining, during the period of time, an amount of processor time at host spent executing the first set of request tasks; and
determining, during the period of time, the request cost index based on the amount of processor time spent.
15. A host computing device in a provider network, the host computing device comprising:
a set of one or more processors;
a main queue;
an overload queue;
a thread pool comprising a plurality of threads; and
a set of instructions which when executed cause the host to:
receive a request to execute a generative operation from another host computing device in the provider network;
execute the generative operation over a time period to yield an operation result;
send the operation result to the other host computing device in the provider network;
dequeue, during the period of time, each request task of a first set of request tasks spawned by executing the generative operation from the main queue;
send, during the period of time, each request task of the first set of request tasks to the thread pool for execution;
determine, during the period of time, a request cost index reflecting an amount of computing resources used to execute the first set of request tasks;
determine, during the period of time, based on the request cost index, to queue, in the overload queue, subsequent request tasks spawned by executing the generative operation;
queue, during the period of time, a second set of request tasks spawned by executing the generative operation in the overload queue; and
dequeue, during the period of time, each request task of the second set of request tasks from the overload queue; and
send, during the period of time, each request task of the second set of request tasks to the thread pool for execution.
16. The system of claim 15 , wherein the set of instructions when executed further cause the host to:
send, during the period of time, based on determining there is an idle thread in the thread pool, a request task spawned by executing the generative operation directly to the thread pool, bypassing the main queue and the overload queue.
17. The system of claim 15 , wherein the set of instructions when executed further cause the host to:
classify, during the period of time, the generative operation as normal or heavy based on the request cost index exceeding a threshold.
18. The system of claim 15 , wherein the set of instructions when executed further cause the host to:
dequeue, during the period of time, the first set of request tasks and send, during the period of time, the first set of request tasks to the thread pool at a first rate; and
dequeue, during the period of time, the second set of request tasks and send, during the period of time, the second set of request tasks to the thread pool at a second rate that is lower than the first rate.
19. The system of claim 15 , wherein the set of instructions when executed further cause the host to:
determine, during the period of time, a set of node request cost indexes for the first set of request tasks; and
determine, during the period of time, the request cost index based on a sum of the set of node request cost indexes.
20. The system of claim 15 , wherein the set of instructions when executed further cause the host to:
determine, during the period of time, a set of node request cost indexes for the first set of request tasks; and
determine, during the period of time, the request cost index based on an average of the set of node request cost indexes.