AUTOMATED SELECTION OF LARGE LANGUAGE MODELS IN CLOUD COMPUTING ENVIRONMENTS
Systems or methods for the selection of large language models (LLMs). A system receives a request from a service that hosts an application. The request is configured to be processed by an LLM to generate a response. The system applies a classification model to the request to determine the class of the request. The classification model is a language model trained to receive text and classify the text into a plurality of classes. The system selects an LLM from a plurality of candidate LLMs based in part on the determined class of the request and recommends the selected LLM to the application.
1 . (canceled)
2 . A method for deploying large language models (LLMs) in a cloud computing environment, comprising:
receiving a request from an application for AI-based processing;
selecting an LLM from a plurality of candidate LLMs based on the LLM having at least a threshold GPU consumption criterion for performing the AI-based processing;
selecting one or more GPU types to be used for deploying the selected LLM;
allocating one or more GPUs of the selected type from a cloud service provider for deploying the selected LLM;
deploying the selected LLM onto the allocated one or more GPUs;
executing the request using the deployed LLM to generate an output; and
providing the output to the application.
3 . The method of claim 2 , wherein the selecting of the LLM is further based on:
applying a classification model to the request to determine a type of the request; and
selecting the LLM based on the determined type.
4 . The method of claim 2 , wherein the selecting of the LLM further comprises:
comparing performance metrics of the plurality of candidate LLMs based on historical request data; and
selecting the LLM based on the comparison.
5 . The method of claim 2 , wherein the selecting of the one or more GPU types is based on historical GPU utilization patterns for each of a plurality of candidate GPU types for the selected LLM.
6 . The method of claim 2 , wherein allocating the one or more GPUs comprises of the selected type:
dividing a GPU into a plurality of virtual GPUs; and
deploying multiple LLMs, including the selected LLM, concurrently on the plurality of virtual GPUs.
7 . The method of claim 2 , further comprising:
determining an input token count for the request; and
determining an output token count for the output generated by the selected LLM.
8 . The method of claim 7 , further comprising:
receiving feedback from a user or application regarding the output generated by the selected LLM; and
in response to receiving a subsequent request from the application, selecting a different LLM based on the feedback.
9 . The method of claim 7 , further comprising:
determining whether the request should be restructured based on the input token count;
responsive to determining that the request should be restructured, reconstructing the request into a new form with a reduced input token count; and
sending the reconstructed request to the selected LLM.
10 . A non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to perform steps comprising:
receiving a request from an application for AI-based processing;
selecting an LLM from a plurality of candidate LLMs based on the LLM having at least a threshold GPU consumption criterion for performing the AI-based processing;
selecting one or more GPU types to be used for deploying the selected LLM;
allocating one or more GPUs of the selected type from a cloud service provider for deploying the selected LLM;
deploying the selected LLM onto the allocated one or more GPUs;
executing the request using the deployed LLM to generate an output; and
providing the output to the application.
11 . The non-transitory computer readable storage medium of claim 10 , wherein the selecting of the LLM is further based on:
applying a classification model to the request to determine a type of the request; and
selecting the LLM based on the determined type.
12 . The non-transitory computer readable storage medium of claim 10 , wherein the selecting of the LLM further comprises:
comparing performance metrics of the plurality of candidate LLMs based on historical request data; and
selecting the LLM based on the comparison.
13 . The non-transitory computer readable storage medium of claim 10 , wherein the selecting of the one or more GPU types is based on historical GPU utilization patterns for each of a plurality of candidate GPU types for the selected LLM.
14 . The non-transitory computer readable storage medium of claim 10 , wherein allocating the one or more GPUs comprises of the selected type:
dividing a GPU into a plurality of virtual GPUs; and
deploying multiple LLMs, including the selected LLM, concurrently on the plurality of virtual GPUs.
15 . The non-transitory computer readable storage medium of claim 10 , further comprising:
determining an input token count for the request; and
determining an output token count for the output generated by the selected LLM.
16 . The non-transitory computer readable storage medium of claim 15 , further comprising:
receiving feedback from a user or application regarding the output generated by the selected LLM; and
in response to receiving a subsequent request from the application, selecting a different LLM based on the feedback.
17 . The non-transitory computer readable storage medium of claim 15 , further comprising:
determining whether the request should be restructured based on the input token count;
responsive to determining that the request should be restructured, reconstructing the request into a new form with a reduced input token count; and
sending the reconstructed request to the selected LLM.
18 . A system, comprising:
one or more processors; and
a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to perform steps comprising:
receiving a request from an application for AI-based processing;
selecting an LLM from a plurality of candidate LLMs based on the LLM having at least a threshold GPU consumption criterion for performing the AI-based processing;
selecting one or more GPU types to be used for deploying the selected LLM;
allocating one or more GPUs of the selected type from a cloud service provider for deploying the selected LLM;
deploying the selected LLM onto the allocated one or more GPUs;
executing the request using the deployed LLM to generate an output; and
providing the output to the application.
19 . The system of claim 18 , wherein the selecting of the LLM is further based on:
applying a classification model to the request to determine a type of the request; and
selecting the LLM based on the determined type.
20 . The system of claim 18 , wherein the selecting of the LLM further comprises:
comparing performance metrics of the plurality of candidate LLMs based on historical request data; and
selecting the LLM based on the comparison.
21 . The system of claim 18 , wherein the selecting of the one or more GPU types is based on historical GPU utilization patterns for each of a plurality of candidate GPU types for the selected LLM.