Global endpoints
Techniques for supporting a token based service are described. In some examples, a token based service such as a foundation model service supports global endpoints. Global endpoints for the token based service are provided with a global quota that is exposed per account, per model. Invocations against a global endpoint consume against the global quota.
1 . A computer-implemented method comprising:
receiving a request from an account to perform an operation using a shared resource at a global endpoint in a first type of region;
determining a second type of region for a pairing of the account and global endpoint, wherein all calls to the shared resource from any first type of region go through the second type of region;
calling the determined second type of region to obtain a quota lease for the pairing of the account and global endpoint, wherein a quota lease is a subset of a global quota for the shared resource;
recording the quota lease in the second type of region;
performing quota enforcement in the first type of region to determine the request can be serviced using the quota lease;
calling a third type of region to perform throttling enforcement for the request to determine the request can be serviced;
generating a routing decision for the request, wherein the routing decision is to route the request to a backend of a plurality of backends;
routing the request;
performing the operation; and
providing a result of the operation.
2 . The computer-implemented method of claim 1 , further comprising:
maintaining concurrency information for the shared resource, tokens per minute (TPM) information, and requests per minute (RPM) information, wherein TPM and RPM are defined per account, per shared resource, per region of a cloud provider network.
3 . The computer-implemented method of claim 1 , further comprising:
caching the determined second type of region in the first type of region.
4 . The computer-implemented method of claim 1 , wherein the shared resource is a foundation model.
5 . The computer-implemented method of claim 4 , wherein the request is an inference request to the foundation model.
6 . The computer-implemented method of claim 1 , wherein the global quota is maintained by at least the second type of region.
7 . The computer-implemented method of claim 1 , wherein the throttling enforcement for the request in the servicing region is concurrency-based.
8 . The computer-implemented method of claim 1 , wherein the quota enforcement in the first type of region to is based on a tokens per minute quota.
9 . The computer-implemented method of claim 1 , wherein the quota enforcement in the first type of region to is based on a requests per minute quota.
10 . The computer-implemented method of claim 1 , wherein the routing decision is based at least in part on one or more of health information, tenancy information, and capacity information for the plurality of backends.
11 . The computer-implemented method of claim 1 , wherein a global endpoint has a plurality of third type of regions.
12 . The computer-implemented method of claim 1 , wherein the request is queued and performing the operation is done when capacity is available.
13 . The computer-implemented method of claim 1 , wherein the third type of region supports heterogenous backends.
14 . The computer-implemented method of claim 1 , wherein the quota lease is for additional quota.
15 . The computer-implemented method of claim 1 , wherein the quota lease is for less quota.
16 . A system comprising:
a first one or more computing devices in a first type of region to support a foundation model service in a multi-tenant provider network;
a second one or more computing devices in a second type of region to support a foundation model service in a multi-tenant provider network;
a third one or more computing devices in a third type of region to support a foundation model service in a multi-tenant provider network;
a fourth one or more computing devices in a fourth region to support a foundation model service in a multi-tenant provider network, wherein the regions are to cooperate to:
receive a request from an account to perform an operation using a shared resource at a global endpoint in a first type of region;
determine a second type of region for determine a second type of region for a pairing of the account and global endpoint, wherein all calls to the shared resource from any first type of region go through the second type of region;
calling the determined second type of region to obtain a quota lease for the pairing of the account and global endpoint, wherein a quota lease is a subset of a global quota for the shared resource;
recording the quota lease in the second type of region;
performing quota enforcement in the first type of region to determine the request can be serviced using the quota lease;
calling a third type of region to perform throttling enforcement for the request to determine the request can be serviced;
generating a routing decision for the request, wherein the routing decision is to route the request to a backend of a plurality of backends;
routing the request;
performing the operation; and
providing a result of the operation.
17 . The system of claim 16 , wherein the shared resource is a foundation model.
18 . A computer-implemented method comprising:
receiving an inference request from an account for a foundation model at a global endpoint in an originating region;
determining a root region for a pairing of the account and global endpoint, wherein all calls to the foundation model from any originating region go through the root region;
calling the determined root region to obtain a quota lease for the pairing of the account and global endpoint, wherein a quota lease is a subset of a global quota for the foundation model;
recording the quota lease in the root region;
performing quota enforcement in the originating region to determine the request can be serviced using the quota lease;
calling a serving region to perform concurrency-based enforcement for the request to determine the request can be serviced;
generating a routing decision for the request, wherein the routing decision is to route the request to a backend of a plurality of backends;
routing the request;
performing the inference using the foundation model; and
providing a result of the inference.
19 . The computer-implemented method of claim 18 , wherein the global quota is replicated across the regions.
20 . The computer-implemented method of claim 18 , wherein a global endpoint has a plurality of serving regions.