Systems and methods for automated communication training
Systems and methods for generating data related to chatbot performance may include receiving a plurality of conversations between a chatbot and chatbot users, providing the plurality of conversations to a machine learning model trained to identify patterns based on the plurality of conversations, identifying patterns within the plurality of conversations based on an output of the machine learning model, displaying, using a graphical user interface (GUI), the identified patterns, and retraining the chatbot based on the identified patterns.
1 . A computer-implemented method for identifying patterns in chatbot conversations and retraining a chatbot based on the identified patterns, the method comprising:
receiving a plurality of conversations between the chatbot and chatbot users;
providing the plurality of conversations to a machine learning model trained to identify patterns based on the plurality of conversations;
identifying patterns within the plurality of conversations based on an output of the machine learning model, the identified patterns including analysis indications representing (i) an autocorrect usage, the autocorrect usage including a ratio of the plurality of conversations between the chatbot and the chatbot users, including at least one auto-corrected word, (ii) a percentage of languages among the plurality of conversations, detected by the chatbot, (iii) a percentage of messages classified as long messages among the plurality of conversations, wherein each of the messages classified as long messages has a length greater than three standard deviations of an average message length, and (iv) potential concepts that meet a potential concept threshold, wherein the potential concepts comprise a pair of words from the plurality of conversations that meet a use threshold;
determining first messages among the plurality of conversations that are associated with unknown intents; displaying, using a graphical user interface (GUI), the identified patterns including each of the analysis indications; and
retraining the chatbot based on the displayed identified patterns including each of the analysis indications and based on at least one of the first messages, to improve performance of the chatbot.
2 . The method of claim 1 , wherein the identified patterns comprise conversation paths, the conversation paths including conversation exit points.
3 . The method of claim 1 , wherein the identified patterns comprise conversation paths, the conversation paths including conversation exit points, the method further comprising:
generating a graphic visually depicting the conversation paths;
providing the graphic to a multimodal large language model for analysis; and
receiving a multimodal large language model output from the multimodal large language model, wherein retraining the chatbot comprises training the chatbot based on the multimodal large language model output.
4 . The method of claim 1 , wherein the analysis indications further represent:
(iv) an entity distribution indicating whether the chatbot is using named entity recognition (NER) attributes as intended; and
(v) a percentage of words unknown to the chatbot; and
wherein the method further comprises generating the analysis indications.
5 . The method of claim 1 , wherein the identified patterns comprise potential concepts that meet a potential concept threshold.
6 . The method of claim 1 , wherein the identified patterns comprise potential concepts that meet a potential concept threshold, the method further comprising:
determining decisions that do not meet an intent confidence, wherein messages associated with the decisions are derived from the plurality of conversations;
displaying, using the GUI, the determined decisions; and
retraining the chatbot further based on at least one of the determined decisions.
7 . A system for identifying patterns in chatbot conversations and retraining a chatbot based on the identified patterns, the system comprising:
a data storage device storing processor-readable instructions; and
a processor operatively connected to the data storage device and configured to execute the instructions to perform operations that include:
receiving a plurality of conversations between the chatbot and chatbot users;
identifying patterns within the plurality of conversations, the identified patterns including analysis indications representing (i) an autocorrect usage, the autocorrect usage including a ratio of the plurality of conversations including at least one auto-corrected word, (ii) a percentage of languages among the plurality of conversations, detected by the chatbot, (iii) a percentage of messages classified as long messages among the plurality of conversations, wherein each of the messages classified as long messages has a length greater than three standard deviations of an average message length, and (iv) potential concepts that meet a potential concept threshold, wherein the potential concepts comprise a pair of words from the plurality of conversations that meet a use threshold;
determining first messages among the plurality of conversations that are associated with unknown intents; displaying the identified patterns including each of the analysis indications; and
retraining the chatbot based on the displayed identified patterns including each of the analysis indications and based on at least one of the first messages, to improve performance of the chatbot.
8 . The system of claim 7 , wherein the identified patterns comprise conversation paths, the conversation paths including conversation exit points; wherein the analysis indications further represent:
(iii) a percentage of languages among the plurality of conversations, detected by the chatbot;
(iv) an entity distribution representing whether the chatbot is using named entity recognition (NER) attributes as intended; and
(v) a percentage of words unknown to the chatbot; and wherein the operations further include generating the analysis indications.
9 . The system of claim 7 , wherein the identified patterns comprise a pair of words from the plurality of conversations that meet a use threshold.
10 . The system of claim 7 , wherein the processor is configured to perform operations that further include:
determining first messages from the plurality of conversations that are associated with unknown intents;
determining decisions that do not meet an intent confidence, wherein messages associated with the decisions are derived from the plurality of conversations;
displaying the first messages and the determined decisions; and
retraining the chatbot based on at least one of the first messages or on at least one of the determined decisions.
11 . The system of claim 7 , wherein the processor is configured to perform operations that further include:
determining decisions that do not meet an intent confidence, wherein messages associated with the decisions are derived from the plurality of conversations;
displaying the determined decisions; and
retraining the chatbot based on at least one of the determined decisions.
12 . A non-transitory computer-readable medium for identifying patterns in chatbot conversations and retraining a chatbot based on the identified patterns, the non-transitory computer-readable medium storing instructions thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a plurality of conversations between the chatbot and chatbot users;
providing the plurality of conversations to a machine learning model trained to identify patterns based on the plurality of conversations;
identifying patterns within the plurality of conversations based on an output of the machine learning model, the identified patterns including analysis indications representing (i) an autocorrect usage, the autocorrect usage including a ratio of the plurality of conversations including at least one auto-corrected word, (ii) a percentage of languages among the plurality of conversations, detected by the chatbot, (iii) a percentage of messages classified as long messages among the plurality of conversations, wherein each of the messages classified as long messages has a length greater than three standard deviations of an average message length, and (iv) potential concepts that meet a potential concept threshold and wherein the potential concepts comprise a pair of words from the plurality of conversations that meet a use threshold;
determining first messages among the plurality of conversations that are associated with unknown intents;
displaying, using a graphical user interface (GUI), the identified patterns including each of the analysis indications; and
retraining the chatbot based on the displayed identified patterns including each of the analysis indications and based on at least one of the first messages, to improve performance of the chatbot.
13 . The non-transitory computer-readable medium of claim 12 , wherein the identified patterns comprise conversation paths, the conversation paths including conversation exit points.
14 . The non-transitory computer-readable medium of claim 12 , wherein the identified patterns comprise conversation paths, the conversation paths including conversation exit points, the operations further comprising:
generating a graphic visually depicting the conversation paths;
providing the graphic to a multimodal large language model for analysis; and
receiving a multimodal large language model output from the multimodal large language model, wherein retraining the chatbot comprises training the chatbot based on the multimodal large language model output.
15 . The non-transitory computer-readable medium of claim 12 , wherein the analysis indications further represent:
(iii) a percentage of languages among the plurality of conversations, detected by the chatbot;
(iv) an entity distribution representing whether the chatbot is using named entity recognition (NER) attributes as intended; and
(v) a percentage of words unknown to the chatbot; and wherein the operations further comprise generating the analysis indications.
16 . The non-transitory computer-readable medium of claim 12 , wherein the identified patterns comprise potential concepts that meet a potential concept threshold.
17 . The non-transitory computer-readable medium of claim 12 , wherein the identified patterns comprise potential concepts that meet a potential concept threshold and wherein the potential concepts comprise a pair of words from the plurality of conversations that meet a use threshold.
18 . The method of claim 1 , wherein displaying the identified patterns comprises generating a graphic visually depicting conversation paths.
19 . The method of claim 1 , wherein displaying the identified patterns further comprises visually distinguishing dialogs to indicate a ratio of chatbot users that exit relative to those that entered.
20 . The method of claim 1 , further comprising filtering the plurality of conversations to remove messages meeting a short message threshold prior to identifying the patterns.