Serverless functional routing for large language model inference service
View Patent ↗A computer-implemented method for serving a large language model (LLM) application via a serverless function router communicative with multiple endpoints that each have a set of subject matter expert models stored thereon is provided. The computer-implemented method includes receiving a prompt, querying a database comprising multiple datasets for an indication as to which one of the multiple datasets has a highest level of similarity with the prompt, recognizing one of the multiple endpoints as having the set of the expert models stored thereon which have a closest match with the one of the multiple datasets and routing the prompt to the one of the multiple endpoints having the set of the expert models stored thereon which have the closest match with the one of the multiple datasets.
1 . A computer-implemented method for serving a large language model (LLM) application, the computer-implemented method comprising:
disposing a serverless function router in communication with multiple discrete servers serving as multiple endpoints;
grouping subject matter expert models into sets of subject matter expert models and storing each set of subject matter expert models on one of the multiple endpoints;
receiving a prompt at the serverless function router;
querying, by the serverless function router, a database comprising multiple datasets for an indication as to which one of the multiple datasets has a highest level of similarity with the prompt;
recognizing, by the serverless function router, one of the multiple endpoints as having the set of the expert models stored thereon which have a closest match with the one of the multiple datasets; and
routing, by the serverless function router, the prompt to the one of the multiple endpoints having the set of the expert models stored thereon which have the closest match with the one of the multiple datasets.
2 . The computer-implemented method according to claim 1 , wherein the database comprises a vector database.
3 . The computer-implemented method according to claim 1 , wherein each of the multiple datasets has a closest match with the subject matter expert models stored on one of the multiple endpoints.
4 . The computer-implemented method according to claim 3 , wherein:
the set of subject matter expert models of a first one of the multiple endpoints is configured to handle prompts relating to medical subject matter,
the set of subject matter expert models of a second one of the multiple endpoints is configured to handle prompts relating to financial subject matter, and
the set of subject matter expert models of a third one of the multiple endpoints is configured to handle prompts relating to technical subject matter.
5 . The computer-implemented method according to claim 3 , wherein each of the sets of subject matter expert models of each of the first, second and third ones of the multiple endpoints comprises one or more foundation models and one or more fine-tuned models.
6 . The computer-implemented method according to claim 1 , wherein the serverless function router is a prompt aware router and the routing comprises prompt aware routing.
7 . A computer program product for serving a large language model (LLM) application, the computer program product comprising one or more computer readable storage media having computer readable program code collectively stored on the one or more computer readable storage media, the computer readable program code being executed by a processor of a computer system to cause the computer system to perform a method comprising:
disposing a serverless function router in communication with multiple discrete servers serving as multiple endpoints;
grouping subject matter expert models into sets of subject matter expert models and storing each set of subject matter expert models on one of the multiple endpoints;
receiving a prompt at the serverless function router;
receiving a prompt at the serverless function router;
querying, by the serverless function router, a database comprising multiple datasets for an indication as to which one of the multiple datasets has a highest level of similarity with the prompt;
recognizing, by the serverless function router, one of the multiple endpoints as having the set of the expert models stored thereon which have a closest match with the one of the multiple datasets; and
routing, by the serverless function router, the prompt to the one of the multiple endpoints having the set of the expert models stored thereon which have the closest match with the one of the multiple datasets.
8 . The computer program product according to claim 7 , wherein the database comprises a vector database.
9 . The computer program product according to claim 7 , wherein each of the multiple datasets has a closest match with the subject matter expert models stored on one of the multiple endpoints.
10 . The computer program product according to claim 9 , wherein:
the set of subject matter expert models of a first one of the multiple endpoints is configured to handle prompts relating to medical subject matter,
the set of subject matter expert models of a second one of the multiple endpoints is configured to handle prompts relating to financial subject matter, and
the set of subject matter expert models of a third one of the multiple endpoints is configured to handle prompts relating to technical subject matter.
11 . The computer program product according to claim 9 , wherein each of the sets of subject matter expert models of each of the first, second and third ones of the multiple endpoints comprises one or more foundation models and one or more fine-tuned models.
12 . The computer program product according to claim 7 , wherein the serverless function router is a prompt aware router and the routing comprises prompt aware routing.
13 . A computing system comprising:
a processor;
a memory coupled to the processor; and
one or more computer readable storage media coupled to the processor, the one or more computer readable storage media collectively containing instructions that are executed by the processor via the memory to implement a method for serving a large language model (LLM) application comprising:
disposing a serverless function router in communication with multiple discrete servers serving as multiple endpoints;
grouping subject matter expert models into sets of subject matter expert models and storing each set of subject matter expert models on one of the multiple endpoints;
receiving a prompt at the serverless function router;
receiving a prompt at the serverless function router;
querying, by the serverless function router, a database comprising multiple datasets for an indication as to which one of the multiple datasets has a highest level of similarity with the prompt;
recognizing, by the serverless function router, one of the multiple endpoints as having the set of the expert models stored thereon which have a closest match with the one of the multiple datasets; and
routing, by the serverless function router, the prompt to the one of the multiple endpoints having the set of the expert models stored thereon which have the closest match with the one of the multiple datasets.
14 . The computing system according to claim 13 , wherein:
the database comprises a vector database, and
each of the multiple datasets has a closest match with the subject matter expert models stored on one of the multiple endpoints.
15 . The computing system according to claim 14 , wherein:
the set of subject matter expert models of a first one of the multiple endpoints is configured to handle prompts relating to medical subject matter,
the set of subject matter expert models of a second one of the multiple endpoints is configured to handle prompts relating to financial subject matter, and
the set of subject matter expert models of a third one of the multiple endpoints is configured to handle prompts relating to technical subject matter.
16 . The computing system according to claim 14 , wherein each of the sets of subject matter expert models of each of the first, second and third ones of the multiple endpoints comprises one or more foundation models and one or more fine-tuned models.
17 . The computing system according to claim 13 , wherein the serverless function router is a prompt aware router and the routing comprises prompt aware routing.