Multi-variable optimization for routing requests to language models
Systems, methods, and devices that relate to routing requests to large language models (LLMs) are disclosed. In one example aspect, the system receives session-specific data elements in response to a request to generate an output using LLMs. The system determines a hierarchy of operational constraints including privacy protocols and performance requirements. Weights for a multi-variable optimization are dynamically updated using the session-specific data elements. The system executes the multi-variable optimization across candidate LLMs that satisfy privacy constraints and optimize performance constraints. Based on the optimization, at least one candidate LLM is selected and the request is routed to it. In response to performance feedback, the system automatically selects a different LLM to improve one constraint, resulting in degradation of another constraint.
1 . One or more non-transitory, computer-readable storage medium comprising instructions recorded thereon, wherein the instructions, when executed by at least one data processor of a system, cause the system to:
receive, in response to a request to generate an output using large language models (LLMs), a plurality of session-specific data elements including prior interaction data, system environment parameters, and computational context values, wherein the plurality of session-specific data elements is updated based on each request and each response;
determine a hierarchy of operational constraints for routing the request to an LLM, the hierarchy of operational constraints comprising a first subset of constraints that comprises privacy and data handling protocols and a second subset of constraints that comprises processing latency thresholds, model response requirements, and resource allocation limitations;
dynamically update weights, using the plurality of session-specific data elements, for a multi-variable optimization;
execute the multi-variable optimization, using the dynamically updated weights, across a plurality of candidate LLMs, wherein each of the plurality of candidate LLMs satisfies the first subset of constraints and optimizes the second subset of constraints such that any further improvement of one constraint in the second subset causes degradation of at least one other constraint in the second subset;
select, based on the multi-variable optimization, at least one candidate LLM for the request;
route the request to the at least one candidate LLM to cause the at least one candidate LLM to generate the output for the request; and
in response to receiving system performance feedback comprising measured runtime performance of the at least one candidate LLM, the system performance feedback relating to at least one constraint in the second subset, automatically select a different LLM from among the plurality of candidate LLMs to improve the at least one constraint, resulting in the degradation of at least one other constraint in the second subset, wherein each constraint in the second subset has a different respective performance metric and a different respective threshold, and wherein the system performance feedback indicates that a respective performance metric for the at least one constraint has deviated beyond a respective threshold for the at least one constraint.
2 . The one or more non-transitory, computer-readable storage medium of claim 1 , wherein the instructions for automatically selecting a different LLM further cause the system to:
retrieve an updated plurality of session-specific data elements;
dynamically update the weights, using the updated plurality of session-specific data elements, for the multi-variable optimization;
increase a weight associated with an objective within the multi-variable optimization corresponding to the at least one constraint in the second subset;
re-execute the multi-variable optimization using the increased weight and the updated plurality of session-specific data elements to generate an updated candidate set of LLMs; and
select, as the different LLM, a candidate LLM from the plurality of candidate LLMs that provides improved performance for the at least one constraint of the second subset and satisfies the first subset of constraints.
3 . The one or more non-transitory, computer-readable storage medium of claim 1 , wherein the instructions for dynamically updating the weights for the multi-variable optimization further cause the system to:
monitor the prior interaction data, the system environment parameters, and the computational context values for the request;
determine revised weights for the multi-variable optimization based on changes in the prior interaction data, the system environment parameters, or the computational context values for the request; and
apply the revised weights in performing the multi-variable optimization to adjust a relative importance of each objective associated with the second subset of constraints.
4 . The one or more non-transitory, computer-readable storage medium of claim 1 , wherein the instructions for selecting, based on the multi-variable optimization, the at least one candidate LLM to process the request further cause the system to:
compare, for each of the plurality of candidate LLMs that satisfy the first subset of constraints, results of the multi-variable optimization with the dynamically updated weights; and
select the at least one candidate LLM that most closely satisfies the second subset of constraints in accordance with the dynamically updated weights.
5 . The one or more non-transitory, computer-readable storage medium of claim 1 , wherein the instructions for receiving the plurality of session-specific data elements further cause the system to, prior to executing the multi-variable optimization, filter the plurality of session-specific data elements to exclude data elements that do not satisfy the privacy and data handling protocols of the first subset of constraints, such that only compliant session-specific data elements are used for dynamically updating the weights for the multi-variable optimization.
6 . The one or more non-transitory, computer-readable storage medium of claim 1 , wherein the instructions further cause the system, prior to routing the request to the at least one candidate LLM, to:
determine a context complexity score derived from the plurality of session-specific data elements; and
apply a complexity threshold for selecting the at least one candidate LLM based on the context complexity score.
7 . A method comprising:
receiving, in response to a request to generate an output using a set of AI models, a plurality of session-specific data elements including prior interaction data, system environment parameters, and computational context values;
determining a first subset of constraints and a second subset of constraints for routing the request to an AI model of the set of AI models, the first subset of constraints comprising privacy and data handling protocols and the second subset of constraints comprising processing latency thresholds, model response requirements, and resource allocation limitations;
updating weights, using the plurality of session-specific data elements, for a multi-variable optimization;
executing the multi-variable optimization, using the updated weights, across a plurality of candidate AI models of the set of AI models, wherein each of the plurality of candidate AI models satisfies the first subset of constraints and optimizes the second subset of constraints;
selecting, based on the multi-variable optimization, at least one candidate AI model for the request;
routing the request to the at least one candidate AI model to cause the at least one candidate AI model to generate the output for the request;
in response to receiving system performance feedback relating to at least one constraint in the second subset, automatically selecting a different AI model from among the plurality of candidate AI models to improve the at least one constraint, wherein each constraint in the second subset has a different respective performance metric and a different respective threshold, and wherein the system performance feedback indicates that a respective performance metric for the at least one constraint has deviated beyond a respective threshold for the at least one constraint.
8 . The method of claim 7 , further comprising, in response to receiving system performance feedback relating to at least one constraint in the second subset, automatically selecting a different AI model from among the plurality of candidate AI models to improve the at least one constraint.
9 . The method of claim 8 , wherein automatically selecting a different AI model further comprises:
retrieving an updated plurality of session-specific data elements;
dynamically updating the weights, using the updated plurality of session-specific data elements, for the multi-variable optimization;
increasing a weight associated with an objective within the multi-variable optimization corresponding to the at least one constraint in the second subset;
re-executing the multi-variable optimization using the increased weight and the updated plurality of session-specific data elements to generate an updated candidate set of AI models; and
selecting, as the different AI model, a candidate AI model from the plurality of candidate AI models that provides improved performance for the at least one constraint of the second subset and satisfies the first subset of constraints.
10 . The method of claim 7 , wherein dynamically updating the weights for the multi-variable optimization further comprises:
monitoring the prior interaction data, the system environment parameters, and the computational context values for the request;
determining revised weights for the multi-variable optimization based on changes in the prior interaction data, the system environment parameters, or the computational context values for the request; and
applying the revised weights in performing the multi-variable optimization to adjust a relative importance of each objective associated with the second subset of constraints.
11 . The method of claim 7 , wherein selecting, based on the multi-variable optimization, the at least one candidate AI model to process the request further comprises:
comparing, for each of the plurality of candidate AI models that satisfy the first subset of constraints, results of the multi-variable optimization with the updated weights; and
selecting the at least one candidate AI model that most closely satisfies the second subset of constraints in accordance with the updated weights.
12 . The method of claim 7 , wherein receiving the plurality of session-specific data elements further comprises, prior to executing the multi-variable optimization, filtering the plurality of session-specific data elements to exclude data elements that do not satisfy the privacy and data handling protocols of the first subset of constraints, such that only compliant session-specific data elements are used for dynamically updating the weights for the multi-variable optimization.
13 . The method of claim 7 , further comprising, prior to routing the request to the at least one candidate AI model:
determining a context complexity score derived from the plurality of session-specific data elements; and
applying a complexity threshold for selecting the at least one candidate AI model based on the context complexity score.
14 . A system comprising:
a storage device; and
one or more processors communicatively coupled to the storage device storing instructions thereon, that cause the one or more processors to:
receive, in response to a request to generate an output using a set of AI models, a plurality of session-specific data elements including prior interaction data, system environment parameters, and computational context values;
determine a first subset of constraints and a second subset of constraints for routing the request to an AI model of the set of AI models, the first subset of constraints comprising privacy and data handling protocols and the second subset of constraints comprising processing latency thresholds, model response requirements, and resource allocation limitations;
update weights, using the plurality of session-specific data elements, for a multi-variable optimization;
execute the multi-variable optimization, using the updated weights, across a plurality of candidate AI models of the set of AI models, wherein each of the plurality of candidate AI models satisfies the first subset of constraints and optimizes the second subset of constraints;
select, based on the multi-variable optimization, at least one candidate AI model for the request;
route the request to the at least one candidate AI model to cause the at least one candidate AI model to generate the output for the request;
in response to receiving system performance feedback relating to at least one constraint in the second subset, automatically select a different AI model from among the plurality of candidate AI models to improve the at least one constraint, wherein each constraint in the second subset has a different respective performance metric and a different respective threshold, and wherein the system performance feedback indicates that a performance metric for the at least one constraint has deviated beyond a respective threshold for the at least one constraint.
15 . The system of claim 14 , wherein the instructions further cause the one or more processors to, in response to receiving system performance feedback relating to at least one constraint in the second subset, automatically select a different AI model from among the plurality of candidate AI models to improve the at least one constraint.
16 . The system of claim 15 , wherein the instructions for automatically selecting a different AI model further cause the one or more processors to:
retrieve an updated plurality of session-specific data elements;
dynamically update the weights, using the updated plurality of session-specific data elements, for the multi-variable optimization;
increase a weight associated with an objective within the multi-variable optimization corresponding to the at least one constraint in the second subset;
re-execute the multi-variable optimization using the increased weight and the updated plurality of session-specific data elements to generate an updated candidate set of AI models; and
select, as the different AI model, a candidate AI model from the plurality of candidate AI models that provides improved performance for the at least one constraint of the second subset and satisfies the first subset of constraints.
17 . The system of claim 14 , wherein the instructions for dynamically updating the weights for the multi-variable optimization further cause the one or more processors to:
monitor the prior interaction data, the system environment parameters, and the computational context values for the request;
determine revised weights for the multi-variable optimization based on changes in the prior interaction data, the system environment parameters, or the computational context values for the request; and
apply the revised weights in performing the multi-variable optimization to adjust a relative importance of each objective associated with the second subset of constraints.
18 . The system of claim 14 , wherein the instructions for selecting, based on the multi-variable optimization, the at least one candidate AI model to process the request further cause the one or more processors to:
compare, for each of the plurality of candidate AI models that satisfy the first subset of constraints, results of the multi-variable optimization with the updated weights; and
select the at least one candidate AI model that most closely satisfies the second subset of constraints in accordance with the updated weights.
19 . The system of claim 14 , wherein the instructions for receiving the plurality of session-specific data elements further cause the one or more processors, prior to executing the multi-variable optimization, to filter the plurality of session-specific data elements to exclude data elements that do not satisfy the privacy and data handling protocols of the first subset of constraints, such that only compliant session-specific data elements are used for dynamically updating the weights for the multi-variable optimization.
20 . The system of claim 14 , wherein the instructions further cause the one or more processors, prior to routing the request to the at least one candidate AI model, to:
determine a context complexity score derived from the plurality of session-specific data elements; and
apply a complexity threshold for selecting the at least one candidate AI model based on the context complexity score.