Multi-stage large language model prompt compression
In an example embodiment, prompt compression is used in two different places in the LLM query pipeline to reduce the size of LLM queries, which reduces the time it takes for an LLM to respond to a query and improves scalability. Specifically, the conversation history itself may be run through a token compressor, which acts to eliminate unnecessary tokens from the conversation history and thus also from a rewritten version of the query. Additionally, the context chunks retrieved during RAG are themselves run through a token compressor to eliminate unnecessary tokens from the context chunks. The compressed context chunks and the rewritten query are then passed to the LLM for processing, and since both the context chunks and the rewritten query contain compressed elements, the result is that the LLM processes fewer tokens to respond to the query.
1 . A system comprising:
at least one hardware processor; and
a computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
accessing a query from a user for a large language model (LLM);
retrieving a conversation history between the user and the LLM from a database;
passing the conversation history through a first token compressor to compress the conversation history;
rewriting the query based on the compressed conversation history;
identifying context chunks in a vector database that are related to the rewritten query;
passing the context chunks through a second token compressor to compress the context chunks;
assembling the compressed context chunks into a single compressed context chunk;
forming a prompt with the rewritten query and the single compressed context chunk;
passing the prompt to the LLM to generate a response to the query; and
receiving the generated response output by the LLM.
2 . The system of claim 1 , wherein the first token compressor and the second token compressor are a single shared token compressor.
3 . The system of claim 2 , wherein the single shared token compressor is a prompt compressor.
4 . The system of claim 1 , wherein the identifying context chunks utilizing embeddings.
5 . The system of claim 1 , wherein the operations further comprise:
using a machine learning model to generate a classification for the query; and
wherein the forming comprises forming the prompt with the rewritten query, the single compressed context chunk, and the classification.
6 . The system of claim 1 , wherein the operations further comprise compressing the prompt using a prompt compressor prior to passing the prompt to the LLM.
7 . The system of claim 5 , wherein the operations further comprise compressing the prompt using a prompt compressor prior to passing the prompt to the LLM.
8 . A method comprising:
accessing a query from a user for a large language model (LLM);
retrieving a conversation history between the user and the LLM from a database;
passing the conversation history through a first token compressor to compress the conversation history;
rewriting the query based on the compressed conversation history;
identifying context chunks in a vector database that are related to the rewritten query;
passing the context chunks through a second token compressor to compress the context chunks;
assembling the compressed context chunks into a single compressed context chunk;
forming a prompt with the rewritten query and the single compressed context chunk;
passing the prompt to the LLM to generate a response to the query; and
receiving the generated response output by the LLM.
9 . The method of claim 8 , wherein the first token compressor and the second token compressor are a single shared token compressor.
10 . The method of claim 9 , wherein the single shared token compressor is a prompt compressor.
11 . The method of claim 8 , wherein the identifying context chunks utilizes embeddings.
12 . The method of claim 8 , further comprising:
using a machine learning model to generate a classification for the query; and
wherein the forming comprises forming the prompt with the rewritten query, the single compressed context chunk, and the classification.
13 . The method of claim 8 , further comprising compressing the prompt using a prompt compressor prior to passing the prompt to the LLM.
14 . The method of claim 12 , further comprising compressing the prompt using a prompt compressor prior to passing the prompt to the LLM.
15 . A non-transitory machine-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
accessing a query from a user for a large language model (LLM);
retrieving a conversation history between the user and the LLM from a database;
passing the conversation history through a first token compressor to compress the conversation history;
rewriting the query based on the compressed conversation history;
identifying context chunks in a vector database that are related to the rewritten query;
passing the context chunks through a second token compressor to compress the context chunks;
assembling the compressed context chunks into a single compressed context chunk;
forming a prompt with the rewritten query and the single compressed context chunk;
passing the prompt to the LLM to generate a response to the query; and
receiving the generated response output by the LLM.
16 . The non-transitory machine-readable medium of claim 15 , wherein the first token compressor and the second token compressor are a single shared token compressor.
17 . The non-transitory machine-readable medium of claim 16 , wherein the single shared token compressor is a prompt compressor.
18 . The non-transitory machine-readable medium of claim 15 , wherein the identifying context chunks utilizing embeddings.
19 . The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
using a machine learning model to generate a classification for the query; and
wherein the forming comprises forming the prompt with the rewritten query, the single compressed context chunk, and the classification.
20 . The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise compressing the prompt using a prompt compressor prior to passing the prompt to the LLM.