Apparatus for routing requests
Apparatus for routing requests from a plurality of connected clients to a plurality of connected servers comprises a processor, memory and a network interface. The processor is configured to run a plurality of identical processes, each being for receiving requests and connecting each received request to a server. For each process, the processor is configured to maintain a queue of requests in memory, determine a number of queued requests that may be connected to a server, and attempt to connect this number of queued requests. The processor then accepts further requests, and if the queue is not empty, places the further requests in the queue, and if the queue is empty, attempts to connect the further requests. The processor determines the number of queued requests that may be connected to a server in dependence upon the length of the queues of all the processes and the number of available connections.
1. Apparatus for routing requests from a plurality of connected clients to a plurality of connected servers, comprising a processor, memory and a network interface, wherein said processor is configured to:
run a plurality of identical processes, each process being for receiving requests and connecting each received request to a server, thereby creating a plurality of connections;
maintain data in memory indicating, for each server, a maximum number of connections it can service, thereby defining a global maximum number of serviceable connections across said plurality of servers, such that at any time the number of available connections is determined by the difference between said global maximum and the number of connections;
and when running one of said processes:
maintain a queue of requests in memory,
determine a number of queued requests that may be connected to a server,
attempt to connect said number of queued requests;
accept further requests, and
if said queue is not empty, place said further requests in said queue, and if said queue is empty, attempt to connect said further requests;
wherein said processor determines said number of queued requests that may be connected to a server in dependence upon the length of the queues of all the processes, queue length of said queue, and the number of available connections.
2. The apparatus of claim 1 , wherein the number of said identical processes is equal to the number of computation units presented by said processor.
3. The apparatus of claim 1 , wherein each of said plurality of identical processes periodically updates data in memory indicating the number of requests in its queue and its number of connections.
4. The apparatus of claim 1 , wherein said processor, when running one of said processes, accepts further requests in batches.
5. The apparatus of claim 4 , wherein said processor, when running one of said processes, determines the size of said batches in dependence upon the number of requests in said queue.
6. The apparatus of claim 1 , wherein said processor, when running one of said processes, is further configured to:
inspect said queued requests for a persistent attribute, said persistent attribute defining a persistent server that must serve a particular request;
identify a request as a persistent request if said request comprises said persistent attribute; and
attempt to connect said persistent request to its persistent server if the number of connections to said server is not in excess of the maximum number of connections it can service.
7. The apparatus of claim 6 , wherein said processor, when running one of said processes, is further configured to leave said persistent request in said queue if said persistent server is unable to service said persistent request.
8. A method of routing requests from a plurality of connected clients to a plurality of connected servers, comprising the steps of:
ascertaining, for each of said plurality of connected servers, a maximum number of connections it can service, thereby defining a global maximum number of serviceable connections, such that at any time the number of available connections is determined by the difference between said global maximum and the number of connections,
receiving requests from said plurality of connected clients,
maintaining a plurality of queues for queuing said requests,
selecting a queue containing queued requests,
determining a number of queued requests that may be connected to a server,
attempting to connect said number of queued requests,
receiving further requests, and
if said queue is not empty, placing said further requests in said queue, and if said queue is empty, attempting to connect said further requests;
wherein said number of queued requests that may be connected to a server is determined in dependence upon the length of each of said plurality of queues, queue length of said queue, and the number of available connections.
9. The method of claim 8 , wherein said step of receiving further requests further comprises the steps of, for each of the queues:
accepting a batch of further requests, and,
placing in said queue said further requests.
10. The method of claim 9 , wherein said step of accepting a batch of further requests further comprises the steps of, for each of the queues:
determining the length of a queue,
accepting a number of further requests determined by the length of the queue.
11. The method of claim 8 , further comprising the steps of:
inspecting said queued requests for a persistent attribute, said persistent attribute defining a persistent server that must serve a particular request;
identifying a request as a persistent request if said request comprises said persistent attribute; and
attempting to connect said persistent request to its persistent server if the number of connections to said server is not in excess of the maximum number of connections it can service.
12. The method of claim 11 , further comprising the step of leaving said persistent request in said queue if said persistent server is unable to service said persistent request.
13. A non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method of routing requests from a plurality of connected clients to a plurality of connected servers, comprising the steps of:
ascertaining, for each of said plurality of connected servers, a maximum number of connections it can service, thereby defining a global maximum number of serviceable connections, such that at any time the number of available connections is determined by the difference between said global maximum and the number of connections,
receiving requests from said plurality of connected clients,
maintaining a plurality of queues for queuing said requests,
selecting a queue containing queued requests,
determining a number of queued requests that may be connected to a server,
attempting to connect said number of queued requests,
receiving further requests, and
if said queue is not empty, placing said further requests in said queue, and if said queue is empty, attempting to connect said further requests;
wherein said number of queued requests that may be connected to a server is determined in dependence upon the length of each of said plurality of queues, queue length of said queue, and the number of available connections.
14. An apparatus, comprising:
one or more processors; and
a non-transitory computer-readable medium storing instructions that, when executed by said one or more processors, cause the apparatus to perform a method of routing requests from a plurality of connected clients to a plurality of connected servers, comprising the steps of:
ascertaining, for each of said plurality of connected servers, a maximum number of connections it can service, thereby defining a global maximum number of serviceable connections, such that at any time the number of available connections is determined by the difference between said global maximum and the number of connections,
receiving requests from said plurality of connected clients,
maintaining a plurality of queues for queuing said requests,
selecting a queue containing queued requests,
determining a number of queued requests that may be connected to a server,
attempting to connect said number of queued requests,
receiving further requests, and
if said queue is not empty, placing said further requests in said queue, and if said queue is empty, attempting to connect said further requests;
wherein said number of queued requests that may be connected to a server is determined in dependence upon the length of each of said plurality of queues, queue length of said queue, and the number of available connections.