Summarizing context and conditioning a large language model to generate responses based on summarized context
The present disclosure relates to systems, non-transitory computer-readable media, and methods for reducing context size for a large language model (LLM) by generating context summaries. In particular, the present disclosure reduces context size and improves the accuracy of existing LLMs by generating and storing context summaries. In particular, the disclosed systems utilize an LLM to determine what context to summarize and when to summarize the context as a conversation progresses. The disclosed systems further update model context to include context summaries instead of full session transcripts or model reasoning. Furthermore, in some implementations, the disclosed systems can use a re-materialization prompt to trigger the LLM to recall particular portions of session transcripts or model reasoning to condition the LLM. Thus, the system can use context summaries to fill context windows for informing LLM response generation as opposed to storing memory-intensive conversation transcripts and model reasoning.
1 . A computer-implemented method comprising:
generating, during an ongoing session between a client device and a large language model, a model context comprising a session transcript for conditioning the large language model for the ongoing session;
generating, using the large language model during the ongoing session, a context summary summarizing a portion of the session transcript from the ongoing session with the large language model;
replacing, within the model context, the portion of the session transcript with the context summary;
receiving, from the client device, a rematerialization query that triggers the large language model to modify the model context by rematerializing the portion of the session transcript corresponding to the context summary;
generating a response to the rematerialization query utilizing the large language model to process the model context comprising the portion of the session transcript rematerialized from the context summary; and
after generating the response to the rematerialization query:
removing the portion of the session transcript from the model context; and
adding the context summary to the model context.
2 . The computer-implemented method of claim 1 , further comprising:
generating a summarization prompt comprising a summarization condition and instructions to generate the context summary; and
using the large language model to generate the context summary based on the summarization prompt.
3 . The computer-implemented method of claim 2 , further comprising determining that the summarization condition is satisfied by at least one of:
determining that a context window for the model context is full;
determining that a token threshold for the model context has been met;
determining that content is older than a threshold age limit; or
determining that a topic for the ongoing session has changed.
4 . The computer-implemented method of claim 1 , further comprising rematerializing the portion of the session transcript by:
inserting the portion of the session transcript into the model context; and
removing the context summary from the model context.
5 . The computer-implemented method of claim 1 , further comprising:
receiving, from the client device, an additional query; and
generating a summary-based response to the additional query utilizing the large language model to process the model context comprising the context summary.
6 . The computer-implemented method of claim 1 , further comprising:
generating context summaries that summarize portions of the session transcript;
associating the context summaries with corresponding portions of the session transcript; and
storing the context summaries and their corresponding portions of the session transcript in a summarization database.
7 . The computer-implemented method of claim 1 , further comprising:
generating, during the ongoing session, a reasoning model context comprising model reasoning for conditioning the large language model;
generating, using the large language model, a reasoning context summary summarizing a portion of the model reasoning from the ongoing session with the large language model;
replacing, within the reasoning model context, the portion of the model reasoning with the reasoning context summary; and
generating a response to a user query utilizing the large language model to process the model context comprising the reasoning context summary.
8 . The computer-implemented method of claim 1 , further comprising:
storing, within a tree-based hierarchical structure within a summarization database, the context summary within a parent node; and
storing, within the tree-based hierarchical structure within the summarization database, the portion of the session transcript within a child node.
9 . The computer-implemented method of claim 1 , further comprising:
generating, using the large language model and during the ongoing session, a meta-summary summarizing one or more context summaries within the model context, wherein the one or more context summaries comprises the context summary;
generating an updated model context by replacing the one or more context summaries with the meta-summary;
receiving, from the client device, a query; and
generating a meta-summary-based response to the query using the large language model to process the updated model context comprising the meta-summary.
10 . The computer-implemented method of claim 1 , further comprising:
receiving, from the client device, a query;
generating a rematerialization prompt instructing the large language model to determine whether the portion of the session transcript is required to respond to the query; and
determining, using the large language model and the rematerialization prompt, that the query qualifies as the rematerialization query.
11 . A system comprising:
at least one processor; and
a non-transitory computer readable medium comprising instructions that, when executed by the at least one processor, cause the system to:
generate, during an ongoing session between a client device and a large language model, a model context comprising a session transcript for conditioning the large language model for the ongoing session;
generate, using the large language model during the ongoing session, a context summary summarizing a portion of the session transcript from the ongoing session with the large language model;
replace, within the model context, the portion of the session transcript with the context summary;
receive, from the client device, a rematerialization query that triggers the large language model to modify the model context by rematerializing the portion of the session transcript corresponding to the context summary;
generate a response to the rematerialization query utilizing the large language model to process the model context comprising the portion of the session transcript rematerialized from the context summary; and
after generating the response to the rematerialization query:
remove the portion of the session transcript from the model context; and
add the context summary to the model context.
12 . The system of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the system to:
receive, from the client device, a query;
generate a rematerialization prompt instructing the large language model to determine whether the portion of the session transcript is required to respond to the query; and
determine, using the large language model and the rematerialization prompt, that the query qualifies as the rematerialization query.
13 . The system of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generate, using the large language model and during the ongoing session, a meta-summary summarizing one or more context summaries within the model context, wherein the one or more context summaries comprises the context summary;
generate an updated model context by replacing the one or more context summaries with the meta-summary;
receive, from the client device, a query; and
generate a meta-summary-based response to the query using the large language model to process the updated model context comprising the meta-summary.
14 . The system of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the system to:
generate a summarization prompt comprising a summarization condition and instructions to generate the context summary; and
use the large language model to generate the context summary based on the summarization prompt.
15 . The system of claim 14 , further comprising instructions that, when executed by the at least one processor, cause the system to determine that the summarization condition is satisfied by at least one of:
determining that a context window for the model context is full;
determining that a token threshold for the model context has been met;
determining that content is older than a threshold age limit; or
determining that a topic for the ongoing session has changed.
16 . A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to:
generate, during an ongoing session between a client device and a large language model, a model context comprising a session transcript for conditioning the large language model for the ongoing session;
generate, using the large language model during the ongoing session, a context summary summarizing a portion of the session transcript from the ongoing session with the large language model;
replace, within the model context, the portion of the session transcript with the context summary;
receive, from the client device, a rematerialization query that triggers the large language model to modify the model context by rematerializing the portion of the session transcript corresponding to the context summary;
generate a response to the rematerialization query utilizing the large language model to process the model context comprising the portion of the session transcript rematerialized from the context summary; and
after generating the response to the rematerialization query:
remove the portion of the session transcript from the model context; and
add the context summary to the model context.
17 . The non-transitory computer readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to rematerialize the portion of the session transcript by:
inserting the portion of the session transcript into the model context; and
removing the context summary from the model context.
18 . The non-transitory computer readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
receive, from the client device, an additional query; and
generate a summary-based response to the additional query utilizing the large language model to process the model context comprising the context summary.
19 . The non-transitory computer readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
generate, during the ongoing session, a reasoning model context comprising model reasoning for conditioning the large language model;
generate, using the large language model, a reasoning context summary summarizing a portion of the model reasoning from the ongoing session with the large language model;
replace, within the reasoning model context, the portion of the model reasoning with the reasoning context summary; and
generate a response to a user query utilizing the large language model to process the model context comprising the reasoning context summary.
20 . The non-transitory computer readable medium of claim 16 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to:
generate context summaries that summarize portions of the session transcript;
associate the context summaries with corresponding portions of the session transcript; and
store the context summaries and their corresponding portions of the session transcript in a summarization database.