Conversational bot evaluation and reinforcement using meaningful automated connection scores
Disclosed embodiments provide a framework to assist bot managers and builders in identifying particular friction points between bots and customers to allow for real-time identification of bot conversation issues and to train bots to improve conversation flows. Conversation data is processed using machine learning models to detect bot states within conversations and calculate a Meaningful Automated Connection Score (MACS) for these conversations. The MACS for a conversation is provided to bot builders to allow the bot builders to identify friction points and update bots accordingly.
1. A computer-implemented method comprising:
receiving conversation data corresponding to a conversation between a conversation bot agent and a customer, wherein the conversation data includes messages exchanged between the conversation bot agent and the customer;
calculating a set of features from the conversation data;
detecting a set of bot states corresponding to performance of the conversation bot agent in the conversation, wherein the set of bot states is detected based on the set of features and the conversation data;
training a machine learning model using sample conversation data and corresponding bot states, wherein training includes determining a Meaningful Automated Connection Score (MACS) for conversations;
providing a MACS for the conversation and the set of bot states;
receiving an update to the conversation bot agent;
implementing the update to the conversation bot agent, wherein the update is used to train the machine learning model;
receiving a new message associated with the conversation;
generating a determination as to whether a new bot state is present within the new message;
dynamically updating the MACS for the conversation based on the determination, the conversation data, and the new message; and
providing the updated MACS.
2. The computer-implemented method of claim 1 , further comprising:
training a bot state machine learning model using the sample conversation data and annotations made to the sample conversation data, wherein training includes determining bot states for conversations; and
using the bot state machine learning model to detect the set of bot states.
3. The computer-implemented method of claim 1 , further comprising:
calculating confidence scores corresponding to a set of possible bot states, wherein the confidence scores are calculated based on the set of features and the conversation data;
applying a threshold to the confidence scores, wherein bot states are detected as a result of corresponding confidence scores being greater than the threshold.
4. The computer-implemented method of claim 1 , further comprising:
extracting conversation lengths corresponding to the sample conversation data; and
using the conversation lengths to further train the machine learning model, wherein the machine learning model is further trained to identify correlations between conversation lengths and the MACS for the conversations.
5. The computer-implemented method of claim 1 , wherein the machine learning model is further trained based on confidence scores corresponding to presence of bot states in the sample conversation data, wherein the confidence scores are obtained from a bot state machine learning model.
6. The computer-implemented method of claim 1 , wherein the machine learning model is further trained using a set of features extracted from the sample conversation data.
7. The computer-implemented method of claim 1 , further comprising:
transferring the conversation from the conversation bot agent to a live agent, wherein the conversation is transferred as a result of updated MACS falling below a threshold value.
8. A system, comprising:
one or more processors; and
memory storing thereon instructions that, as a result of being executed by the one or more processors, cause the system to:
receive conversation data corresponding to a conversation between a conversation bot agent and a customer, wherein the conversation data includes messages exchanged between the conversation bot agent and the customer;
calculate a set of features from the conversation data;
detect a set of bot states corresponding to performance of the conversation bot agent in the conversation, wherein the set of bot states is detected based on the set of features and the conversation data;
train a machine learning model using sample conversation data and corresponding bot states, wherein training includes determining a Meaningful Automated Connection Score (MACS) for conversations;
provide a MACS for the conversation and the set of bot states;
receive an update to the conversation bot agent; and
implement the update to the conversation bot agent, wherein the update is used to train the machine learning model;
receive a new message associated with the conversation;
generate a determination as to whether a new bot state is present within the new message;
dynamically update the MACS for the conversation, wherein the MACS is updated based on the determination, the conversation data, and the new message; and
provide the updated MACS.
9. The system of claim 8 , wherein the instructions that cause the system to detect the set of bot states further cause the system to:
train a bot state machine learning model using the sample conversation data and annotations made to the sample conversation data, wherein training includes determining bot states for conversations; and
use the bot state machine learning model to detect the set of bot states.
10. The system of claim 9 , wherein the instructions further cause the system to:
transfer the conversation from the conversation bot agent to a live agent, wherein the conversation is transferred as a result of updated MACS falling below a threshold value.
11. The system of claim 8 , wherein the instructions that cause the system to detect the set of bot states further cause the system to:
calculate confidence scores corresponding to a set of possible bot states, wherein the confidence scores are calculated based on the set of features and the conversation data; and
apply a threshold to the confidence scores, wherein bot states are detected as a result of corresponding confidence scores being greater than the threshold.
12. The system of claim 8 , wherein the instructions further cause the system to:
extract conversation lengths corresponding to the sample conversation data; and
use the conversation lengths to further train the machine learning model, wherein the machine learning model is further trained to identify correlations between conversation lengths and the MACS for the conversations.
13. The system of claim 8 , wherein the machine learning model is further trained based on confidence scores corresponding to presence of bot states in the sample conversation data, wherein the confidence scores are obtained from a bot state machine learning model.
14. The system of claim 8 , wherein the machine learning model is further trained using a set of features extracted from the sample conversation data.
15. A non-transitory, computer-readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to:
receive conversation data corresponding to a conversation between a conversation bot agent and a customer, wherein the conversation data includes messages exchanged between the conversation bot agent and the customer;
calculate a set of features from the conversation data;
detect a set of bot states corresponding to performance of the conversation bot agent in the conversation, wherein the set of bot states is detected based on the set of features and the conversation data;
train a machine learning model using sample conversation data and corresponding bot states, wherein training includes determining a Meaningful Automated Connection Score (MACS) for conversations;
provide a MACS for the conversation and the set of bot states;
receive an update to the conversation bot agent; and
implement the update to the conversation bot agent, wherein the update is used to train the machine learning model;
receive a new message associated with the conversation;
generate a determination as to whether a new bot state is present within the new message;
dynamically update the MACS for the conversation, wherein the MACS is updated based on the determination, the conversation data, and the new message; and
provide the updated MACS.
16. The non-transitory, computer-readable medium of claim 15 , wherein the executable instructions that cause the computer system to detect the set of bot states further cause the computer system to:
train a bot state machine learning model using the sample conversation data and annotations made to the sample conversation data, wherein training includes determining bot states for conversations; and
use the bot state machine learning model to detect the set of bot states.
17. The non-transitory, computer-readable medium of claim 15 , wherein the executable instructions that cause the computer system to detect the set of bot states further cause the computer system to:
calculate confidence scores corresponding to a set of possible bot states, wherein the confidence scores are calculated based on the set of features and the conversation data; and
apply a threshold to the confidence scores, wherein bot states are detected as a result of corresponding confidence scores being greater than the threshold.
18. The non-transitory, computer-readable medium of claim 15 , wherein the executable instructions further cause the computer system to:
extract conversation lengths corresponding to the sample conversation data; and
use the conversation lengths to further train the machine learning model, wherein the machine learning model is further trained to identify correlations between conversation lengths and the MACS for the conversations.
19. The non-transitory, computer-readable medium of claim 15 , wherein the machine learning model is further trained based on confidence scores corresponding to presence of bot states in the sample conversation data, wherein the confidence scores are obtained from a bot state machine learning model.
20. The non-transitory, computer-readable storage medium of claim 15 , wherein the machine learning model is further trained using a set of features extracted from the sample conversation data.
21. The non-transitory, computer-readable medium of claim 15 , wherein the executable instructions further cause the computer system to:
transfer the conversation from the conversation bot agent to a live agent, wherein the conversation is transferred as a result of updated MACS falling below a threshold value.