Method for large language model selection and query routing via offline quality estimation and token-efficent context optimization

1 . A method comprising:

accessing a user query characterized by an initial quantity of tokens;

deriving a set of candidate transforms for the user query by:

detecting a first set of language signals in the user query, the first set of language signals comprising a first subset of language signals characterized by a first quantity of tokens; and

based on semantic alignment between the first subset of language signals and a second subset of language signals, deriving a first candidate transform, in the set of candidate transforms, for the first subset of language signals that yields the second subset of language signals characterized by a second quantity of tokens, each candidate transform in the set of candidate transforms characterized by an amount of token reduction to the initial quantity of tokens, the first candidate transform characterized by a first amount of token reduction corresponding to a difference between the first quantity of tokens and the second quantity of tokens;

accessing a quality prediction model that predicts quality scores of responses from a set of language models according to queries;

for each language model in the set of language models:

calculating a predicted quality score, in a set of predicted quality scores, of a projected response by the language model based on the user query and the quality prediction model; and

calculating a resource estimate, in a set of resource estimates, for the projected response by the language model by calculating an estimated cost, in a set of estimated costs, for the projected response by the language model based on:

the initial quantity of tokens; and

an estimated quantity of tokens in the projected response;

selecting a target language model, in the set of language models, for the user query based on:

the set of predicted quality scores;

the set of resource estimates comprising the set of estimated costs; and

amounts of token reduction, characterizing the set of candidate transforms, comprising the first amount of token reduction;

deriving a target combination of transforms in the set of candidate transforms based on the target language model;

generating a target prompt based on the user query and the target combination of transforms, the target prompt characterized by a final quantity of tokens falling below the initial quantity of tokens; and

generating an output specifying:

the target language model; and

the target prompt.

2 . The method of claim 1 , further comprising:

accessing a first reference query, in a set of reference queries, representing a first software development task;

accessing a first reference response in a set of reference responses, the first reference response:

corresponding to the first reference query; and

comprising a set of reference program code that resolves the first software development task;

for a first language model in the set of language models:

invoking the first language model to generate a first test response based on the first reference query;

calculating a first set of differences between the first test response and the first reference response; and

calculating a first test quality score, in a first set of test quality scores for the first language model, based on the first set of differences; and

training the quality prediction model to predict quality scores of responses by the first language model based on the set of reference queries and the first set of test quality scores.

3 . The method of claim 2 :

wherein accessing the first reference response comprises accessing the first reference response comprising the reference set of program code characterized by a first quantity of tokens;

wherein invoking the first language model to generate the first test response comprises invoking the first language model to generate the first test response comprising a first set of program code characterized by a second quantity of tokens; and

wherein calculating the first test quality score comprises calculating the first test quality score based on:

a difference between the first quantity of tokens and the second quantity of tokens; and

a confidence that the first set of program code resolves the first software development task.

4 . The method of claim 2 :

wherein accessing the first reference query comprises:

accessing the first reference query characterized by a first quantity of tokens;

deriving a first combination of transforms, for the first reference query, that reduces the first quantity of tokens; and

generating a first transformed reference query based on the first reference query and the first combination of transforms, the first transformed reference query characterized by a second quantity of tokens falling below the first quantity of tokens;

wherein invoking the first language model to generate the first test response based on the first reference query comprises:

invoking the first language model to generate the first test response according to the first reference query; and

invoking the first language model to generate a second test response according to the first transformed reference query;

further comprising calculating a second set of differences between the second test response and the first reference response;

wherein calculating the first test quality score comprises:

calculating the first test quality score, in a first subset of test quality scores in the first set of test quality scores, based on the first set of differences; and

calculating a second test quality score, in a second subset of test quality scores in the first set of test quality scores, based on the second set of differences; and

wherein training the quality prediction model comprises training the quality prediction model to predict quality scores of responses by the first language model based on:

the first subset of test quality scores according to a first weight; and

the second subset of test quality scores according to a second weight falling below the first weight.

5 . The method of claim 2 :

further comprising, for a second language model in the set of language models:

invoking the second language model to generate a second test response based on the first reference query;

calculating a second set of differences between the first test response and the first reference response; and

calculating a second test quality score, in a second set of test quality scores for the second language model, based on the second set of differences; and

wherein training the quality prediction model comprises training the quality prediction model to predict quality scores of responses by the second language model based on the set of reference queries and the second set of test quality scores.

6 . The method of claim 1 :

wherein deriving the first candidate transform comprises:

calculating a first context score representing semantic alignment between the first subset of language signals and the second subset of language signals; and

in response to the first context score exceeding a context score threshold, deriving the first candidate transform for the first subset of language signals that yields the second subset of language signals.

7 . The method of claim 1 :

wherein deriving the first candidate transform comprises:

calculating a first quality impact score representing reduction of predicted quality of the projected response based on replacement of the first subset of language signals with the second subset of language signals; and

in response to the first quality impact score falling below a threshold quality impact score, deriving the first candidate transform for the first subset of language signals that yields the second subset of language signals.

8 . The method of claim 1 :

further comprising:

accessing a policy defining:

a threshold quality score;

a threshold cost; and

a threshold response time;

wherein calculating the resource estimate for the projected response for each language model comprises:

calculating an estimated response time, in a set of estimated response times, for the projected response by the language model based on the user query; and

wherein selecting the target language model for the user query comprises selecting a first language model in the set of language models for the user query based on:

a first predicted quality score, in the set of predicted quality scores, associated with the first language model and exceeding the threshold quality score;

a first estimated cost, in the set of estimated costs, associated with the first language model and falling below the threshold cost; and

a first estimated response time, in the set of estimated response times, associated with the first language model and falling below the threshold response time.

9 . The method of claim 8 , wherein selecting the target language model for the user query comprises:

for each language model in the set of language models, calculating an aggregate score, in a set of aggregate scores, of the language model based on:

the predicted quality score of the projected response by the language model;

the resource estimate for the projected response by the language model; and

the estimated response time for the projected response by the language model; and

selecting the target language model characterized by a greatest aggregate score in the set of aggregate scores.

10 . The method of claim 8 :

wherein selecting the target language model for the user query comprises:

accessing an objective function representing:

a first weight for predicted quality scores in the set of predicted quality scores;

a second weight for estimated costs in the set of estimated costs; and

a third weight for estimated response times in the set of estimated response times; and

selecting the target language model for the user query based on:

the set of predicted quality scores;

the set of estimated costs;

amounts of token reduction characterizing the set of candidate transforms; and

the objective function.

11 . The method of claim 1 :

wherein deriving the target combination of transforms comprises deriving a target permutation of transforms, in the set of candidate transforms, compatible with the target language model; and

wherein generating the target prompt comprises generating the target prompt by sequentially transforming the user query into the target prompt according to the target permutation of transforms.

12 . The method of claim 1 , wherein generating the output comprises:

generating a visualization specifying:

an identifier of the target language model;

the target prompt;

a predicted quality score, in the set of predicted quality scores, associated with the target language model; and

a resource estimate, in the set of resource estimates, associated with the target language model; and

serving the visualization to a user.

13 . The method of claim 1 , wherein generating the output comprises:

invoking the target language model to generate a response for the user query according to the target prompt; and

serving the response to a user.

14 . The method of claim 13 :

wherein invoking the target language model comprises exclusively invoking the target language model to generate the response for the user query absent invocations to other language models, in the set of language models, for the user query.

15 . A method comprising:

accessing a set of reference queries representing software development tasks;

accessing a set of reference responses corresponding to the set of reference queries;

for each language model in the set of language models:

for each reference query in the set of reference queries:

invoking the language model to generate a test response based on the reference query;

calculating a set of differences between the test response and a reference response, in the set of reference responses, corresponding to the reference query; and

calculating a test quality score, in a set of test quality scores for the language model, based on the set of differences; and

based on the set of reference queries and the set of test quality scores, training a quality prediction model to predict quality scores of projected responses by the language model according to queries;

accessing a user query representing a software development task;

for each language model in the set of language models:

calculating a predicted quality score, in a set of predicted quality scores, of a projected response by the language model based on the user query and the quality prediction model; and

calculating a resource estimate, in a set of resource estimates, for the projected response by the language model;

selecting a target language model, in the set of language models, for the user query based on:

the set of predicted quality scores; and

the set of resource estimates;

generating a visualization specifying:

an identifier of the target language model;

a predicted quality score, in the set of predicated quality scores, associated with the target language model; and

a resource estimate, in the set of resource estimates, associated with the target language model; and

serving the visualization to a user.

16 . The method of claim 15 :

wherein accessing the user query comprises accessing the user query characterized by an initial quantity of tokens;

further comprising deriving a set of candidate transforms for the user query, each candidate transform in the set of candidate transforms characterized by an amount of token reduction to the initial quantity of tokens; and

wherein selecting the target language model for the user query comprises selecting the target language model for the user query based on:

the set of predicted quality scores;

the set of resource estimates; and

amounts of token reduction characterizing the set of candidate transforms.

17 . The method of claim 16 , further comprising:

deriving a combination of transforms, in the set of transforms, compatible with the target language model;

generating a target prompt based on the user query and the combination of transforms;

in response to confirmation of the target language model from the user, invoking the target model to generate a response for the user query according to the target prompt; and

serving the response to a user.

18 . A method comprising:

accessing a first reference query, in a set of reference queries, representing a first software development task;

accessing a first reference response in a set of reference responses, the first reference response:

corresponding to the first reference query; and

comprising a set of reference program code that resolves the first software development task;

for a first language model in a set of language models:

invoking the first language model to generate a first test response based on the first reference query;

calculating a first set of differences between the first test response and the first reference response; and

calculating a first test quality score, in a first set of test quality scores for the first language model, based on the first set of differences;

training a quality prediction model to predict quality scores of responses by the first language model based on the set of reference queries and the first set of test quality scores;

accessing a user query characterized by an initial quantity of tokens;

deriving a set of candidate transforms for the user query, each candidate transform in the set of candidate transforms characterized by an amount of token reduction to the initial quantity of tokens;

accessing the quality prediction model that predicts quality scores of responses from the set of language models according to queries;

for each language model in the set of language models:

calculating a predicted quality score, in a set of predicted quality scores, of a projected response by the language model based on the user query and the quality prediction model; and

calculating a resource estimate, in a set of resource estimates, for the projected response by the language model;

selecting a target language model, in the set of language models, for the user query based on:

the set of predicted quality scores;

the set of resource estimates; and

amounts of token reduction characterizing the set of candidate transforms;

deriving a target combination of transforms in the set of candidate transforms based on the target language model;

generating a target prompt based on the user query and the target combination of transforms, the target prompt characterized by a final quantity of tokens falling below the initial quantity of tokens; and

generating an output specifying:

the target language model; and

the target prompt.

19 . A method comprising:

accessing a policy defining:

a threshold quality score;

a threshold cost; and

a threshold response time;

accessing a user query characterized by an initial quantity of tokens;

deriving a set of candidate transforms for the user query, each candidate transform in the set of candidate transforms characterized by an amount of token reduction to the initial quantity of tokens;

accessing a quality prediction model that predicts quality scores of responses from a set of language models according to queries;

for each language model in the set of language models:

calculating a predicted quality score, in a set of predicted quality scores, of a projected response by the language model based on the user query and the quality prediction model; and

calculating a resource estimate, in a set of resource estimates, for the projected response by the language model by:

calculating an estimated cost, in a set of estimated costs, for the projected response by the language model based on the user query; and

calculating an estimated response time, in a set of estimated response times, for the projected response by the language model based on the user query;

selecting a first language model, in the set of language models, for the user query based on:

a first predicted quality score, in the set of predicted quality scores, associated with the first language model and exceeding the threshold quality score;

a first estimated cost, in the set of estimated costs, associated with the first language model and falling below the threshold cost;

a first estimated response time, in the set of estimated response times, associated with the first language model and falling below the threshold response time; and

amounts of token reduction characterizing the set of candidate transforms;

deriving a target combination of transforms in the set of candidate transforms based on the first language model;

generating a target prompt based on the user query and the target combination of transforms, the target prompt characterized by a final quantity of tokens falling below the initial quantity of tokens; and

generating an output specifying:

the first language model; and

the target prompt.