Query replay for personalized responses in an LLM powered assistant

1 . A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:

receiving a natural language query from a user that solicits a response from an assistant large language model (LLM);

processing the natural language query to identify, from a log of refinement queries each previously input by the user for instructing the assistant LLM to refine a corresponding previous response generated by the assistant LLM, a preferred refinement query that is relevant to the natural language query;

prompting, using the natural language query and the preferred refinement query identified as being relevant to the natural language query, the assistant LLM to generate a personalized response to the natural language query; and

providing, for output from a user device, the personalized response to the natural language query,

wherein prompting the assistant LLM to generate the personalized response to the natural language query comprises:

structuring the preferred refinement query into a refinement prompt, the refinement prompt specifically formulated for the assistant LLM to personalize base responses generated by the assistant LLM;

structuring a composite prompt by concatenating the natural language query with the refinement prompt; and

issuing, for input to the assistant LLM, the composite prompt to cause the assistant LLM to:

generate a base response to the natural language query, the base response not personalized for the user; and

augment the base response to convey personal preferences specified by the refinement prompt, wherein the personalized response comprises the augmented base response.

2 . The computer-implemented method of claim 1 , wherein receiving the natural language query comprises:

receiving audio data characterizing an utterance of the natural language query spoken by the user and captured by the user device; and

performing speech recognition on the audio data to generate a textual representation of the natural language query spoken by the user.

3 . The computer-implemented method of claim 1 , wherein providing the personalized response to the natural language query comprises providing, for output from the user device, response content comprising both the personalized response and the base response to the natural language query.

4 . The computer-implemented method of claim 1 , wherein structuring the preferred refinement query into the refinement prompt comprises structuring the preferred refinement query into the refinement prompt by appending natural language personalized response instructions to a textual representation of the preferred refinement query.

5 . The computer-implemented method of claim 1 , wherein augmenting the base response comprises at least one of:

highlighting information or results in the base response;

annotating text in the base response;

underlining or modifying text in the base response; or

adjusting a formatting of the base response.

6 . The computer-implemented method of claim 1 , wherein:

each corresponding refinement query in the log of refinement queries is paired with a respective previous query embedding of a corresponding previous query input by the user that prompted the assistant LLM to generate the corresponding previous response that was refined by the corresponding refinement query, the respective previous query embedding of each corresponding previous query projected into an embedding space; and

processing the natural language query to identify the preferred refinement query that is relevant to the natural language query comprises:

embedding, using a neural network, the natural language query into a query embedding;

identifying, from the previous query embeddings projected into the embedding space, a closest one of the previous query embeddings to the query embedding within the embedding space; and

selecting the refinement query in the log of refinement queries that is paired with the identified closest one of the previous query embeddings as the preferred refinement query.

7 . The computer-implemented method of claim 1 , wherein the operations further comprise:

receiving local context associated with the natural language query,

wherein structuring the composite prompt by concatenating the natural language query with the refinement prompt further comprises structuring the composite prompt by concatenating the natural language query with the refinement prompt and the local context.

8 . The computer-implemented method of claim 7 , wherein:

the natural language query comprises text; and

the local context is concatenated in plain text with the natural language query.

9 . The computer-implemented method of claim 7 , wherein the local context comprises at least one of:

a recent activity history comprising previous queries input by the user and corresponding responses generated by the assistant LLM during a dialog session;

geographical location data;

site visits;

recent documents from a private corpus; or

recent user history information associated with the natural language query.

10 . A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:

receiving a natural language query from a user that solicits a response from an assistant large language model (LLM);

processing the natural language query to identify, from a log of refinement queries each previously input by the user for instructing the assistant LLM to refine a corresponding previous response generated by the assistant LLM, a preferred refinement query that is relevant to the natural language query;

prompting, using the natural language query and the preferred refinement query identified as being relevant to the natural language query, the assistant LLM to generate a personalized response to the natural language query; and

providing, for output from a user device, the personalized response to the natural language query,

wherein:

each corresponding refinement query in the log of refinement queries comprises a respective confidence value assigned to the corresponding refinement query; and

prompting the assistant LLM to generate the personalized response to the natural language query comprises:

determining the respective confidence value threshold assigned to the preferred refinement query satisfies a confidence threshold;

based on determining the respective confidence value assigned to the preferred refinement query satisfies the confidence value threshold, structuring the natural language query and the preferred refinement query into a natural language composite prompt, the natural language composite prompt specifically formulated for instructing the assistant LLM to generate the personalized response to the natural language query; and

issuing, for input to the assistant LLM, the natural language composite prompt to cause the assistant LLM to generate the personalized response to the natural language query.

11 . The computer-implemented method of claim 10 , wherein the natural language composite prompt comprises a single sentence.

12 . A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving a natural language query from a user that solicits a response from an assistant large language model (LLM);

processing the natural language query to identify, from a log of refinement queries each previously input by the user for instructing the assistant LLM to refine a corresponding previous response generated by the assistant LLM, a preferred refinement query that is relevant to the natural language query;

prompting, using the natural language query and the preferred refinement query identified as being relevant to the natural language query, the assistant LLM to generate a personalized response to the natural language query; and

providing, for output from a user device, the personalized response to the natural language query,

wherein prompting the assistant LLM to generate the personalized response to the natural language query comprises:

structuring the preferred refinement query into a refinement prompt, the refinement prompt specifically formulated for the assistant LLM to personalize base responses generated by the assistant LLM;

structuring a composite prompt by concatenating the natural language query with the refinement prompt; and

issuing, for input to the assistant LLM, the composite prompt to cause the assistant LLM to:

generate a base response to the natural language query, the base response not personalized for the user; and

augment the base response to convey personal preferences specified by the refinement prompt, wherein the personalized response comprises the augmented base response.

13 . The system of claim 12 , wherein receiving the natural language query comprises:

receiving audio data characterizing an utterance of the natural language query spoken by the user and captured by the user device; and

performing speech recognition on the audio data to generate a textual representation of the natural language query spoken by the user.

14 . The system of claim 12 , wherein providing the personalized response to the natural language query comprises providing, for output from the user device, response content comprising both the personalized response and the base response to the natural language query.

15 . The system of claim 12 , wherein structuring the preferred refinement query into the refinement prompt comprises structuring the preferred refinement query into the refinement prompt by appending natural language personalized response instructions to a textual representation of the preferred refinement query.

16 . The system of claim 12 , wherein augmenting the base response comprises at least one of:

highlighting information or results in the base response;

annotating text in the base response;

underlining or modifying text in the base response; or

adjusting a formatting of the base response.

17 . The system of claim 12 , wherein:

each corresponding refinement query in the log of refinement queries is paired with a respective previous query embedding of a corresponding previous query input by the user that prompted the assistant LLM to generate the corresponding previous response that was refined by the corresponding refinement query, the respective previous query embedding of each corresponding previous query projected into an embedding space; and

processing the natural language query to identify the preferred refinement query that is relevant to the natural language query comprises:

embedding, using a neural network, the natural language query into a query embedding;

identifying, from the previous query embeddings projected into the embedding space, a closest one of the previous query embeddings to the query embedding within the embedding space; and

selecting the refinement query in the log of refinement queries that is paired with the identified closest one of the previous query embeddings as the preferred refinement query.

18 . The system of claim 12 , wherein the operations further comprise:

receiving local context associated with the natural language query,

wherein structuring the composite prompt by concatenating the natural language query with the refinement prompt further comprises structuring the composite prompt by concatenating the natural language query with the refinement prompt and the local context.

19 . The system of claim 18 , wherein:

the natural language query comprises text; and

the local context is concatenated in plain text with the natural language query.

20 . The system of claim 19 , wherein the local context comprises at least one of:

a recent activity history comprising previous queries input by the user and corresponding responses generated by the assistant LLM during a dialog session;

geographical location data;

site visits;

recent documents from a private corpus; or

recent user history information associated with the natural language query.

21 . A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving a natural language query from a user that solicits a response from an assistant large language model (LLM);

processing the natural language query to identify, from a log of refinement queries each previously input by the user for instructing the assistant LLM to refine a corresponding previous response generated by the assistant LLM, a preferred refinement query that is relevant to the natural language query;

prompting, using the natural language query and the preferred refinement query identified as being relevant to the natural language query, the assistant LLM to generate a personalized response to the natural language query; and

providing, for output from a user device, the personalized response to the natural language query,

wherein each corresponding refinement query in the log of refinement queries comprises a respective confidence value assigned to the corresponding refinement query, and

wherein prompting the assistant LLM to generate the personalized response to the natural language query comprises:

determining the respective confidence value threshold assigned to the preferred refinement query satisfies a confidence threshold;

based on determining the respective confidence value assigned to the preferred refinement query satisfies the confidence value threshold, structuring the natural language query and the preferred refinement query into a natural language composite prompt, the natural language composite prompt specifically formulated for instructing the assistant LLM to generate the personalized response to the natural language query; and

issuing, for input to the assistant LLM, the natural language composite prompt to cause the assistant LLM to generate the personalized response to the natural language query.

22 . The system of claim 21 , wherein the natural language composite prompt comprises a single sentence.