Hybrid inference system for COGS reduction
A hybrid inference system for a coding assistant utilizes a routing model to predict whether output generated by a large language model for a given prompt would be accepted by a user of the coding assistant. The routing model routes the prompt when the routing model indicates that the output generated by the large language model is likely to be accepted. The routing model routes the prompt to a local model when the output generated by the large language model is not likely to be accepted. The routing model is trained on the historical output generated by the large language model for various prompts and the acceptance or rejection of the output by users of the coding assistant.
1 . A system comprising:
a processor; and
a memory that stores a program configured to be executed by the processor, the program comprising instructions that when executed by the processor perform acts that:
obtain, from a user interface of a coding assistant, a prompt to a large language model (LLM) for the LLM to generate a code generation candidate, wherein the LLM resides on an external host;
determine, from a routing model, whether the LLM will generate a low-return output not accepted by a user;
upon the routing model determining that the LLM will generate the low-return output: route the prompt to a local model in proximity to the coding assistant;
receive the code generation candidate from the local model; and
return the code generation candidate to the user interface; and
upon the routing model determining that the LLM will not generate the low-return output:
route the prompt to the LLM;
receive the code generation candidate from the LLM; and
return the code generation candidate to the user interface.
2 . The system of claim 1 , wherein the local model is stored in a computing device of the coding assistant.
3 . The system of claim 1 , wherein the routing model is stored in a computing device of a user of the coding assistant.
4 . The system of claim 1 , wherein the user interface is part of a software development environment.
5 . The system of claim 1 , wherein the routing model is trained on historical data comprising output generated by the LLM for a respective prompt and acceptance/rejection of the output by a respective user of the coding assistant.
6 . The system of claim 1 , wherein the routing model comprises an encoder neural transformer model with attention.
7 . The system of claim 1 , wherein the local model is a neural transformer model with attention.
8 . The system of claim 1 , wherein the large language model is a neural transformer model with attention.
9 . A computer-implemented method, comprising:
receiving, from a user interface of a coding assistant, a prompt to a large language model (LLM), wherein the LLM resides on an external host, wherein the prompt comprises a context for generation of a code generation candidate;
determining, from a routing model, whether the prompt is a low-return prompt that will result in the code generation candidate not accepted by a user of the coding assistant;
upon the routing model determining that the prompt is a low-return prompt: routing the prompt to a local model in proximity of the coding assistant;
causing the local model to generate the code generation candidate given the prompt;
receiving the code generation candidate from the local model; and
returning the code generation candidate to the user interface; and
upon determining that the prompt is not a low-return prompt: routing the prompt to the LLM.
10 . The computer-implemented method of claim 9 , wherein the local model is a neural transformer model with attention.
11 . The computer-implemented method of claim 9 , further comprising:
receiving the code generation candidate from the LLM; and
returning the code generation candidate to the user interface.
12 . The computer-implemented method of claim 9 , wherein the routing model is trained on telemetry data of the LLM, wherein the telemetry data comprises a plurality of training prompts and associated user responses to output of the LLM given a training prompt.
13 . The computer-implemented method of claim 9 , wherein the routing model is trained on historical acceptance of candidates output from the large language model.
14 . The computer-implemented method of claim 9 , wherein the routing model is trained on historical rejections of candidates output from the large language model.
15 . The computer-implemented method of claim 9 , wherein the routing model is an encoder neural transformer model with attention.
16 . A hardware device having stored thereon computer executable instructions that are structured to be executable by a processor of a computing device to thereby cause the computing device to perform actions that:
host a local neural model on the computing device to interact with a coding assistant;
access a large language model to generate an output given a prompt;
obtain a target prompt from a user interface of the coding assistant;
determine from a routing model whether a target output from the large language model given the target prompt has a high probability of being accepted by a user of the coding assistant, wherein the determination is based on historical acceptances and rejections of historical outputs generated by the large language model given different prompts;
upon the routing model indicating that the target output from the large language model given the target prompt has a high probability of being accepted by the user of the coding assistant:
route the prompt to the large language model;
obtain a response from the large language model; and
display the response in the user interface; and
upon the routing model indicating that the target output from the large language model given the target prompt does not have a high probability of being accepted by the user of the coding assistant, route the target prompt to the local neural model.
17 . The hardware device of claim 16 , wherein the routing model is pre-trained on natural language text and source code and fine-tuned on telemetry data of the large language model.
18 . The hardware device of claim 16 having stored thereon computer executable instructions that are structured to be executable by a processor of a computing device to thereby cause the computing device to perform actions that:
utilize a binary classifier as the routing model to determine to predict whether the output from the large language model given the target prompt is likely to be has a high probability of being accepted by the user of the coding assistant.
19 . The hardware device of claim 16 , wherein the binary classifier resides on the computing device of the coding assistant.
20 . The hardware device of claim 16 , wherein the binary classifier is a neural transformer model with attention.