ASSESSING AND IMPROVING THE DEPLOYMENT OF LARGE LANGUAGE MODELS IN SPECIFIC DOMAINS
Techniques are described herein for a method of generating a synthetic chat between a customer module and an agent module, wherein: the customer module receives a first prompt and determines a first chat response, and the agent module receives a second prompt and determines a second chat response; generating, by a summarizer module, a summary of the synthetic chat; scoring, by a scorer module, the synthetic chat by comparing the summary of the synthetic chat to the first prompt and the second prompt; adjusting, based on the score, a parameter associated with the synthetic chat.
1 . (canceled)
2 . A method comprising:
iteratively performing the following:
generating a synthetic chat using an agent large language model (LLM) and a customer LLM;
adjusting, based on the synthetic chat, one or more parameters associated with the synthetic chat to improve performance of the agent LLM for use in a specific domain;
receiving, from a user of a user computing device, a first audio message comprising user speech;
converting the user speech into natural language text;
computing, by the agent LLM with the adjusted one or more parameters, a chat response based on the natural language text;
converting the chat response into a second audio message comprising a synthetic voice; and
causing communication of the second audio message comprising the synthetic voice to the user of the user computing device.
3 . The method of claim 2 , wherein the adjusting is based on linguistic properties comprising one or more of tone, complexity, nuance, domain-specific words, domain-specific phrases, corporate policies, brand, and length of response.
4 . The method of claim 3 , wherein the linguistic properties comprise a plurality of properties, and wherein each of the plurality of properties is scored.
5 . The method of claim 4 , wherein a score for each synthetic chat is based on a weight assigned to each of the plurality of properties.
6 . The method of claim 2 , wherein the adjusting comprising updating a prompt provided to the agent LLM.
7 . The method of claim 2 , wherein the adjusting comprising updating weights and/or hyperparameters of the agent LLM.
8 . The method of claim 2 , wherein each synthetic chat comprises at least a communication generated using the agent LLM and an agent prompt and a communication generated using the customer LLM and a customer prompt.
9 . The method of claim 8 , wherein a score of each synthetic chat is based on a comparison of a summary of the synthetic chat to at least one of the agent prompt or the customer prompt.
10 . The method of claim 2 , wherein the adjusting is based on identifying from the synthetic chat whether a category of information was shared as part of the synthetic chat.
11 . The method of claim 10 , wherein the identifying includes prompting a third LLM regarding whether the synthetic chat included information with the category of information.
12 . The method of claim 2 , wherein the iteratively performing is performed until at least one of the synthetic chats is scored above a threshold.
13 . A non-transitory machine-readable storage medium that provides instructions that, if executed by a system, are configurable to cause said system to perform operations comprising:
iteratively performing the following:
generating a synthetic chat using an agent large language model (LLM) and a customer LLM;
adjusting, based on the synthetic chat, one or more parameters associated with the synthetic chat to improve performance of the agent LLM for use in a specific domain;
receiving, from a user of a user computing device, a first audio message comprising user speech;
converting the user speech into natural language text;
computing, by the agent LLM with the adjusted one or more parameters, a chat response based on the natural language text;
converting the chat response into a second audio message comprising a synthetic voice; and
causing communication of the second audio message comprising the synthetic voice to the user of the user computing device.
14 . The non-transitory machine-readable storage medium of claim 13 , wherein the adjusting is based on linguistic properties comprising one or more of tone, complexity, nuance, domain-specific words, domain-specific phrases, corporate policies, brand, and length of response.
15 . The non-transitory machine-readable storage medium of claim 14 , wherein the linguistic properties comprise a plurality of properties, and wherein each of the plurality of properties is scored.
16 . The non-transitory machine-readable storage medium of claim 15 , wherein a score for each synthetic chat is based on a weight assigned to each of the plurality of properties.
17 . The non-transitory machine-readable storage medium of claim 13 , wherein the adjusting comprising updating a prompt provided to the agent LLM.
18 . The non-transitory machine-readable storage medium of claim 13 , wherein the adjusting comprising updating weights and/or hyperparameters of the agent LLM.
19 . The non-transitory machine-readable storage medium of claim 13 , wherein each synthetic chat comprises at least a communication generated using the agent LLM and an agent prompt and a communication generated using the customer LLM and a customer prompt.
20 . The non-transitory machine-readable storage medium of claim 19 , wherein a score of each synthetic chat is based on a comparison of a summary of the synthetic chat to at least one of the agent prompt or the customer prompt.
21 . The non-transitory machine-readable storage medium of claim 13 , wherein the adjusting is based on identifying from the synthetic chat whether a category of information was shared as part of the synthetic chat.