Summarizing chat sessions to manage information loss

1 . A system comprising:

a processor; and

a memory that stores a program that is configured to be executed by the processor, wherein the program comprises instructions to perform acts that:

obtain a user query for a machine learning model to perform a task, wherein the machine learning model is limited to a fixed-size context window;

maintain, in the memory, a chat history with the machine learning model, wherein the chat history comprises an ordered sequence of prompts and model responses;

in response to determining that a size of the user query and the chat history exceeds the fixed-size context window:

generate a summarized chat history;

replace, in the memory, the chat history with the summarized chat history; and

preserve, in a store separate from the memory, original content of the summarized chat history;

create a first prompt, wherein the first prompt comprises the summarized chat history and the user query;

transmit the first prompt to the machine learning model;

receive from the machine learning model a first response to the first prompt; and

output the first response in a user interface.

2 . The system of claim 1 , wherein the program comprises further instructions to perform acts that:

store the first prompt and the first response in the chat history.

3 . The system of claim 1 , wherein the program comprises further instructions to perform acts that:

invoke the machine learning model to generate the summarized chat history.

4 . The system of claim 1 , wherein the program comprises further instructions to perform acts that:

store the original content of the summarized chat history in an indexed table, wherein the store is indexed by keywords associated with the original content or embeddings associated with the user query.

5 . The system of claim 4 , wherein the keywords represent commonly-used words found in the original content of the summarized chat history.

6 . The system of claim 1 , wherein the program includes further instructions to perform acts that:

obtain a subsequent user query; and

generate a second user prompt comprising an instruction, the subsequent user query and the summarized chat history, wherein the instruction indicates how the machine learning model is to request original content of the summarized chat history.

7 . The system of claim 6 , wherein the program comprises further instructions to perform acts that:

receive a second model response to the second user prompt comprising the requested original content of the summarized chat history and at least one keyword describing the requested original content of the summarized chat history.

8 . The system of claim 7 , wherein the program comprises further instructions to perform acts that:

extract the requested original content of the summarized chat history from the store using the at least one keyword.

9 . The system of claim 8 , wherein the program comprises further instructions to perform acts that:

generate a third user prompt comprising the requested original content of the summarized chat history extracted from a lookup table and a second user query.

10 . A computer-implemented method comprising:

monitoring a chat session with a machine learning model, wherein the chat session comprises an ordered sequence of user prompts followed by model responses, wherein the machine learning model has a fixed-size context window;

recording a chat history of the chat session in a memory, wherein the chat history comprises the user prompts and corresponding model responses;

receiving a user query for the machine learning model to perform a task;

upon determining that a size of the user query and the chat history exceeds the fixed-size context window, summarizing the chat history into a summarized chat history, replacing the user prompts and the corresponding model responses of the chat history with the summarized chat history in the memory, and storing original content of the summarized chat history in a select memory store;

generating a first user prompt for the machine learning model to respond to the user query, wherein the first user prompt comprises the user query and the summarized chat history;

transmitting the first user prompt to the machine learning model;

receiving from the machine learning model a model response to the first user prompt; and

displaying the model response in a user interface.

11 . The computer-implemented method of claim 10 , further comprising:

storing the first user prompt and the model response in the chat history.

12 . The computer-implemented method of claim 10 , further comprising:

generating a second user prompt for the machine learning model to respond to a second user query, wherein the second user prompt comprises an instruction, the second user query and the summarized chat history, wherein the instruction indicates how the machine learning model requests original content of the summarized chat history.

13 . The computer-implemented method of claim 12 , further comprising:

receiving a second model response to the second user prompt, wherein the second model response contains at least one keyword associated with the requested original content of the summarized chat history.

14 . The computer-implemented method of claim 13 , further comprising:

obtaining the requested original content of the summarized chat history from the select memory store using the at least one keyword.

15 . The computer-implemented method of claim 14 , further comprising:

generating a third user prompt comprising the requested original content of the summarized chat history, the second user query, and the chat history.

16 . A hardware storage device having stored thereon computer executable instructions that are structured to be executable by a processor of a computing device to thereby cause the computing device to perform actions that:

facilitate a chat session with a machine learning model, wherein the chat session comprises a plurality of user prompts and model responses, wherein each user prompt is followed by a corresponding model response, wherein the machine learning model has a fixed-size token-constraint;

record the plurality of user prompts and model responses of the chat session in a chat history of a memory;

obtain a first user query for the machine learning model to perform a task;

upon determination of a token size of the first user query and the chat history exceeding the fixed-size token-constraint of the machine learning model, summarize the chat history into a summarized chat history, replace the plurality of user prompts and model responses of the chat session with the summarized chat history in the memory, and store original content of the summarized chat history in an indexed table;

generate a first user prompt for the machine learning model to respond to the first user query, wherein the first user prompt comprises the first user query and the summarized chat history;

transmit the first user prompt to the machine learning model;

receive from the machine learning model a model response to the first user prompt; and

display the model response in a user interface.

17 . The hardware storage device of claim 16 having stored thereon computer executable instructions that are structured to be executable by the processor of the computing device to thereby cause the computing device to perform actions that:

invoke the machine learning model to generate the summarized chat history given the plurality of user prompts and model responses in the chat history.

18 . The hardware storage device of claim 16 , having stored thereon computer executable instructions that are structured to be executable by the processor of the computing device to thereby cause the computing device to perform actions that:

generate a second user prompt for the machine learning model to respond to a second user query, wherein the second user prompt comprises an instruction, the second user query, and the chat history, wherein the instruction indicates how the machine learning model requests original content of the summarized chat history.

19 . The hardware storage device of claim 18 , having stored thereon computer executable instructions that are structured to be executable by the processor of the computing device to thereby cause the computing device to perform actions that:

receive a second model response to the second user prompt, wherein the second model response includes a keyword associated with requested original content of the summarized chat history; and

extract the requested original content of the summarized chat history using the keyword in the indexed table.

20 . The hardware storage device of claim 19 , having stored thereon computer executable instructions that are structured to be executable by the processor of the computing device to thereby cause the computing device to perform actions that:

generate a third user prompt including the requested original content of the summarized chat history, the second user query, and the chat history.