Systems and methods for using contextual interim responses in conversations managed by a virtual assistant server
A virtual assistant server executes a dialog flow associated with a use case determined from a user utterance received from a user device to generate an actual response to the user utterance. The virtual assistant server provides a prompt to a large language model (LLM) comprising: one or more instructions to the LLM to generate a plurality of interim responses to be transmitted to the user device, and input data required by the LLM to generate the plurality of interim responses. The virtual assistant server transmits one or more of the plurality of interim responses received from the LLM to the user device in an order determined by the LLM until the actual response is generated. Subsequently, the virtual assistant server transmits the generated actual response to the user utterance to the user device after the one or more of the plurality of interim responses are transmitted to the user device.
1 . A method comprising:
executing, by a virtual assistant server, a dialog flow comprising a series of interconnected nodes including at least one of: an intent node, one or more entity nodes, one or more invoke-LLM nodes, one or more service nodes, one or more confirmation nodes, or one or more message nodes associated with a use case determined from a user input received from a user device associated with a user as part of an online conversation between the user and a virtual assistant to generate an actual response to the user input;
providing, by the virtual assistant server, a prompt to a large language model (LLM) comprising:
one or more instructions to the LLM to generate a plurality of interim responses to be transmitted to the user device, wherein the plurality of interim responses are generated by the LLM using a probabilistic approach to language generation based on patterns learned from training data comprising input-output pairs of prompts and expected interim responses; and
input data required by the LLM to generate the plurality of interim responses, wherein the input data comprises: a description of the use case, an identification of a current node of execution of the dialog flow, and a description of an action performed at the current node of execution;
transmitting, by the virtual assistant server, one or more of the plurality of interim responses received from the LLM to the user device in an order determined by the LLM prior to generating the actual response; and
transmitting, by the virtual assistant server, the generated actual response to the user input to the user device subsequent to the transmitting the one or more of the plurality of interim responses to the user device.
2 . The method of claim 1 , wherein the input data further comprises: a conversation transcript, a conversation state, and a user sentiment.
3 . The method of claim 1 , wherein the conversation state is a currently executed one of a plurality of interconnected nodes of the dialog flow of the use case during the online conversation.
4 . The method of claim 1 , further comprising:
monitoring, by the virtual assistant server, a first time period beginning immediately after receipt of the user input from the user device and for the generation of the actual response; and
wherein the transmitting the one or more of the interim responses received from the LLM to the user device is based on the monitored first time period and the monitored generation of the actual response.
5 . The method of claim 4 , wherein a first one of the interim responses received from the LLM is transmitted to the user device when the monitored first time period is beyond a first threshold time period and the monitored generation of the actual response is incomplete.
6 . The method of claim 5 , wherein a second one of the interim responses received from the LLM is transmitted to the user device when a monitored second time period is beyond a second threshold time period and the monitored generation of the actual response is incomplete, wherein the monitored second time period begins immediately after the transmitting of the first one of the interim responses to the user device.
7 . The method of claim 1 , wherein the order determined by the LLM is based on execution steps to be performed in a currently executed conversation state of the dialog flow.
8 . The method of claim 1 , wherein the order determined by the LLM is based on an empathy level of each of the plurality of interim responses.
9 . A virtual assistant server comprising:
one or more processors; and
a memory coupled to the one or more processors which are configured to execute programmed instructions stored in the memory to:
execute a dialog flow comprising a series of interconnected nodes including at least one of: an intent node, one or more entity nodes, one or more invoke-LLM nodes, one or more service nodes, one or more confirmation nodes, or one or more message nodes associated with a use case determined from a user input received from a user device associated with a user as part of an online conversation between the user and a virtual assistant to generate an actual response to the user input;
provide a prompt to a large language model (LLM) comprising:
one or more instructions to the LLM to generate a plurality of interim responses to be transmitted to the user device, wherein the plurality of interim responses are generated by the LLM using a probabilistic approach to language generation based on patterns learned from training data comprising input-output pairs of prompts and expected interim responses; and
input data required by the LLM to generate the plurality of interim responses, wherein the input data comprises: a description of the use case, an identification of a current node of execution of the dialog flow, and a description of an action performed at the current node of execution;
transmit one or more of the plurality of interim responses received from the LLM to the user device in an order determined by the LLM prior to generating the actual response; and
transmit the generated actual response to the user input to the user device subsequent to the transmitting the one or more of the plurality of interim responses to the user device.
10 . The virtual assistant server of claim 9 , wherein the input data further comprises: a conversation transcript, a conversation state, and a user sentiment.
11 . The virtual assistant server of claim 9 , wherein the conversation state is a currently executed one of a plurality of interconnected nodes of the dialog flow of the use case during the online conversation.
12 . The virtual assistant server of claim 9 , the one or more processors are further configured to:
monitor a first time period beginning immediately after receipt of the user input from the user device and for the generation of the actual response; and
wherein the transmitting the one or more of the interim responses received from the LLM to the user device is based on the monitored first time period and the monitored generation of the actual response.
13 . The virtual assistant server of claim 12 , wherein a first one of the interim responses received from the LLM is transmitted to the user device when the monitored first time period is beyond a first threshold time period and the monitored generation of the actual response is incomplete.
14 . The virtual assistant server of claim 13 , wherein a second one of the interim responses received from the LLM is transmitted to the user device when a monitored second time period is beyond a second threshold time period and the monitored generation of the actual response is incomplete, wherein the monitored second time period begins immediately after the transmitting of the first one of the interim responses to the user device.
15 . The virtual assistant server of claim 9 , wherein the order determined by the LLM is based on execution steps to be performed in a currently executed conversation state of the dialog flow.
16 . The virtual assistant server of claim 9 , wherein the order determined by the LLM is based on an empathy level of each of the plurality of interim responses.
17 . A non-transitory computer-readable medium storing instructions which when executed by one or more processors, causes the one or more processors to:
execute a dialog flow comprising a series of interconnected nodes including at least one of: an intent node, one or more entity nodes, one or more invoke-LLM nodes, one or more service nodes, one or more confirmation nodes, or one or more message nodes associated with a use case determined from a user input received from a user device associated with a user as part of an online conversation between the user and a virtual assistant to generate an actual response to the user input;
provide a prompt to a large language model (LLM) comprising:
one or more instructions to the LLM to generate a plurality of interim responses to be transmitted to the user device, wherein the plurality of interim responses are generated by the LLM using a probabilistic approach to language generation based on patterns learned from training data comprising input-output pairs of prompts and expected interim responses; and
input data required by the LLM to generate the plurality of interim responses, wherein the input data comprises: a description of the use case, an identification of a current node of execution of the dialog flow, and a description of an action performed at the current node of execution;
transmit one or more of the plurality of interim responses received from the LLM to the user device in an order determined by the LLM prior to generating the actual response; and
transmit the generated actual response to the user input to the user device subsequent to the transmitting the one or more of the plurality of interim responses to the user device.
18 . The non-transitory computer-readable medium of claim 17 , wherein the input data further comprises: a conversation transcript, a conversation state, and a user sentiment.
19 . The non-transitory computer-readable medium of claim 17 , wherein the conversation state is a currently executed one of a plurality of interconnected nodes of the dialog flow of the use case during the online conversation.
20 . The non-transitory computer-readable medium of claim 17 , further comprising instructions which when executed by the one or more processors, causes the one or more processors to:
monitor a first time period beginning immediately after receipt of the user input from the user device and for the generation of the actual response; and
wherein the transmitting the one or more of the interim responses received from the LLM to the user device is based on the monitored first time period and the monitored generation of the actual response.
21 . The non-transitory computer-readable medium of claim 20 , wherein a first one of the interim responses received from the LLM is transmitted to the user device when the monitored first time period is beyond a first threshold time period and the monitored generation of the actual response is incomplete.
22 . The non-transitory computer-readable medium of claim 21 , wherein a second one of the interim responses received from the LLM is transmitted to the user device when a monitored second time period is beyond a second threshold time period and the monitored generation of the actual response is incomplete, wherein the monitored second time period begins immediately after the transmitting of the first one of the interim responses to the user device.
23 . The non-transitory computer-readable medium of claim 17 , wherein the order determined by the LLM is based on execution steps to be performed in a currently executed conversation state of the dialog flow.
24 . The non-transitory computer-readable medium of claim 17 , wherein the order determined by the LLM is based on an empathy level of each of the plurality of interim responses.