Automated selection of large language models in cloud computing environments
Systems or methods for the selection of large language models (LLMs). A system receives a request from a service that hosts an application. The request is configured to be processed by an LLM to generate a response. The system applies a classification model to the request to determine the class of the request. The classification model is a language model trained to receive text and classify the text into a plurality of classes. The system selects an LLM from a plurality of candidate LLMs based in part on the determined class of the request and recommends the selected LLM to the application.
1. A method for selection of Large Language Models (LLMs), the method comprising:
receiving a request from a service that hosts an application, wherein the request is configured to be processed by an LLM to generate a response;
applying a classification model to the request to determine a class of the request, wherein the classification model is trained to receive data examples and classify the data examples into a plurality of classes;
selecting an LLM from a plurality of candidate LLMs based at least in part on the determined class of the request; and
recommending the selected LLM to the application,
wherein the plurality of candidate LLMs comprise a first LLM hosted on an open source platform having a first user feedback score derived from the determined class of the request, and a second LLM hosted on a non-open source platform having a second feedback score derived from the determined class of the request,
wherein the selecting the LLM from the plurality of candidate LLMs comprises:
for a difference between the first user feedback score and the second user feedback score being less than a threshold, selecting the first LLM as the LLM, and
for the difference being equal to or greater than the threshold, selecting the second LLM as the LLM.
2. The method of claim 1 , wherein the classification model is a zero-shot classification model trained to classify requests into classes.
3. The method of claim 1 , further comprising:
determining a number of tokens required for at least one of the plurality of candidate LLMs to process the request; and
selecting the LLM from the plurality of LLMs further based on the number of tokens.
4. The method of claim 1 , further comprising: for the second LLM being selected as the LLM, performing:
determining a GPU utilization rate at a private Kubernetes network that hosts the second LLM;
responsive to determining that the GPU utilization rate is lower than a predetermined threshold,
causing the GPU to be divided into a plurality of virtual GPUs; and
causing each of the plurality of virtual GPUs to execute a separate instance of the second LLM.
5. The method of claim 1 , further comprising:
causing the request to be sent to the selected LLM;
receiving a response from the selected LLM; and
passing the response to the application.
6. The method of claim 5 , wherein the request comprises a first prompt in a first format, the method further comprising:
constructing a second prompt in a second format; and
sending the second prompt to the selected LLM.
7. The method of claim 6 , further comprising:
applying a similarity model to the first prompt to determine similarities between the first prompt and a plurality of historical prompts received from applications;
identifying a set of historical prompts that are similar to the first prompt;
identifying a prompt schema associated with the set of historical prompts; and
applying the prompt schema to the first prompt to generate the second prompt that corresponds to a lower token count than the first prompt.
8. The method of claim 7 , wherein applying the prompt schema to the first prompt comprises:
generating an SQL query that references data in the first prompt, and
integrating the SQL query into the schema to generate the second prompt.
9. A non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to:
receive a request from a service that hosts an application, where the request is configured to be processed by an LLM to generate a response;
apply a classification model to the request to determine a class of the request, wherein the classification model trained to receive examples and classify the examples into a plurality of classes;
select an LLM from a plurality of candidate LLMs;
recommend the selected LLM to the application,
wherein the plurality of candidate LLMs comprise a first LLM hosted on an open source platform having a first user feedback score derived from the determined class of the request, and a second LLM hosted on a non-open source platform having a second feedback score derived from the determined class of the request,
wherein the selecting the LLM from the plurality of candidate LLMs comprises:
for a difference between the first user feedback score and the second user feedback score being less than a threshold, selecting the first LLM as the LLM, and
for the difference being equal to or greater than the threshold, selecting the second LLM as the LLM.
10. The non-transitory computer readable storage medium of claim 9 , wherein the classification model is a zero-shot classification model trained to classify requests into classes.
11. The non-transitory computer readable storage medium of claim 9 , the one or more processors are further caused to:
determine a number of tokens required for at least one of the plurality of candidate LLMs to process the request; and
select the LLM from the plurality of LLMs further based on the number of tokens.
12. The non-transitory computer readable storage medium of claim 9 , wherein the one or more processors are further caused to:
for the selected LLM being non-open-source, perform:
determine a GPU utilization rate at a private Kubernetes network that host the second LLM;
responsive to determining that the GPU utilization rate is lower than a predetermined threshold,
cause the GPU to be divided into a plurality of virtual GPUs; and
cause each of the plurality of virtual GPUs to execute a separate instance of the second LLM.
13. The non-transitory computer readable storage medium of claim 12 , wherein the one or more processors are further caused to:
cause the request to be sent to the selected LLM;
receive a response from the selected LLM; and
pass the response to the application.
14. The non-transitory computer readable storage medium of claim 13 , wherein the request comprises a first prompt in a first format, and the one or more processors are further configured to:
construct a second prompt in a second format; and
send the second prompt to the selected LLM.
15. The non-transitory computer readable storage medium of claim 14 , wherein the one or more processors are further configured to:
apply a similarity model to the first prompt to determine similarities between the first prompt and a plurality of historical prompts received from applications;
identify a set of historical prompts that are similar to the first prompt;
identify a prompt schema associated with the set of historical prompts; and
applying the prompt schema to the first prompt to generate the second prompt that corresponds to a lower token count than the first prompt.
16. The non-transitory computer readable storage medium of claim 15 , wherein applying the prompt schema to the first prompt comprises:
generating an SQL query that references data in the first prompt, and
integrating the SQL query into the schema to generate the second prompt.
17. A computing system, comprising:
one or more processors; and
a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to:
receive a request from a service that hosts an application, where the request is configured to be processed by an LLM to generate a response;
apply a classification model to the request to determine a class of the request, wherein the classification model is trained to receive examples and classify the examples into a plurality of classes;
select an LLM from a plurality of candidate LLMs based at least in part on the determined class of the request; and
recommend the selected LLM to the application,
wherein the plurality of candidate LLMs comprise a first LLM hosted on an open source platform having a first user feedback score derived from the determined class of the request, and a second LLM hosted on a non-open source platform having a second feedback score derived from the determined class of the request,
wherein the selecting the LLM from the plurality of candidate LLMs comprises:
for a difference between the first user feedback score and the second user feedback score being less than a threshold, selecting the first LLM as the LLM, and
for the difference being equal to or greater than the threshold, selecting the second LLM as the LLM.
18. The computing system of claim 17 , wherein the classification model is a zero-shot classification model trained to classify requests into classes.