Systems and methods for query creation for use in automated quality management
Systems and methods for improving query creation and AQM are provided. A user interface is provided that allows an analyst to generate and test queries. After the analyst provides a query, one or more subsets of previous call center interactions are selected to test the provided query using methods such as a large language model or string vectorization. Once the subsets of interactions are selected, an LLM is used to evaluate the query against the subset. The evaluations may indicate interactions which were positive and which interactions were negative and may include quotes from the interactions that support the evaluations. The evaluations are presented to the analyst for review, and the analyst may revise the query in view of the evaluations. Once the analyst is satisfied with the performance of the query, the query may be used by the LLM to perform AQM for new interactions in the call center.
1 . A method comprising:
receiving, by a computing device, a query from a user;
receiving, by the computing device, an identification of a corpus of interactions;
based on the query, selecting, by the computing device, a first sub-corpus of interactions from the corpus of interactions, wherein the interactions in the first sub-corpus are interactions that are relevant to the query and the first sub-corpus of interactions includes fewer interactions than the corpus of interactions;
based on the query, selecting, by the computing device, a second sub-corpus of interactions from the corpus of interactions, wherein the interactions in the second sub-corpus are interactions that are irrelevant to the query and the second sub-corpus of interactions includes fewer interactions than the corpus of interactions;
running, by the computing device, a large language model using the query, the first sub-corpus and the second sub-corpus to identify interactions that satisfy the query and interactions that do not satisfy the query;
presenting, by the computing device, indications of the identified interactions that satisfy the query and the identified interactions that do not satisfy the query to the user; and
after presenting the indications of the identified interactions that satisfy the query and the identified interactions that do not satisfy the query, receiving, by the computing device, a revised query from the user.
2 . The method of claim 1 , wherein the interactions comprise interaction between agents and callers.
3 . The method of claim 1 , further comprising:
running the large language model using the revised query, the first sub-corpus and the second sub-corpus to identify interactions that satisfy the revised query and interactions that do not satisfy the revised query; and
presenting indications of the identified interactions that satisfy the revised query and the identified interactions that do not satisfy the revised query to the user.
4 . The method of claim 1 , further comprising providing a user interface; and receiving the query through the user interface.
5 . The method of claim 4 , further comprising:
presenting a plurality of existing queries to the user through the user interface; and
receiving the query from the user comprises receiving a selection of the query from the plurality of existing queries.
6 . The method of claim 1 , further comprising:
receiving a new interaction, wherein the new interaction is not in the corpus of interactions; and
running the large language model using the query on the received new interaction to provide automated quality management on the received new interaction.
7 . The method of claim 1 , wherein selecting the first sub-corpus comprises:
using the large language model to generate a plurality of words and phrases that are related to the query;
determining interactions in the corpus of interactions that match at least some of the plurality of words and phrases; and
selecting the determined interactions in the corpus of interactions that match at least some of the plurality of words and phrases for the first sub-corpus.
8 . The method of claim 7 , wherein selecting the second sub-corpus comprises:
determining interactions in the corpus of interactions that do not match any of the plurality of words and phrases; and
selecting the determined interactions in the corpus of interactions that do not match any of the plurality of words and phrases for the second sub-corpus.
9 . The method of claim 1 , wherein selecting the first sub-corpus comprises:
vectorizing the query;
vectorizing each phrase of a plurality of phrases from each interaction of the corpus of interactions;
for each interaction of the corpus of interactions, determine a similarity of each vectorized phrase with the vectorized query; and
selecting the interactions from the corpus of interactions for the first sub-corpus based on the determined similarities.
10 . The method of claim 1 , wherein selecting the first sub-corpus comprises randomly selecting a number of interactions from the corpus of interactions.
11 . A system comprising:
a computing device; and
a computer-readable medium with computer-executable instructions stored thereon that when executed by the computing device cause the computing device to:
receive a query from a user;
receive an identification of a corpus of interactions;
based on the query, select a first sub-corpus of interactions from the corpus of interactions, wherein the interactions in the first sub-corpus are interactions that are relevant to the query and the first sub-corpus of interactions includes fewer interactions than the corpus of interactions;
based on the query, select a second sub-corpus of interactions from the corpus of interactions, wherein the interactions in the second sub-corpus are interactions that are irrelevant to the query and the second sub-corpus of interactions includes fewer interactions than the corpus of interactions;
run a large language model using the query, the first sub-corpus and the second sub-corpus to identify interactions that satisfy the query and interactions that do not satisfy the query;
present indications of the identified interactions that satisfy the query and the identified interactions that do not satisfy the query to the user; and
after presenting the indications of the identified interactions that satisfy the query and the identified interactions that do not satisfy the query, receive a revised query from the user.
12 . The system of claim 11 , wherein the interactions comprise interaction between agents and callers.
13 . The system of claim 11 , further comprising computer-executable instructions that when executed by the computing device cause the computing device to:
run the large language model using the revised query, the first sub-corpus and the second sub-corpus to identify interactions that satisfy the revised query and interactions that do not satisfy the revised query; and
present indications of the identified interactions that satisfy the revised query and the identified interactions that do not satisfy the revised query to the user.
14 . The system of claim 11 , further comprising providing a user interface; and receiving the query through the user interface.
15 . The system of claim 14 , further comprising computer-executable instructions that when executed by the computing device cause the computing device to:
present a plurality of existing queries to the user through the user interface; and
receiving a selection of the query from the plurality of existing queries.
16 . The system of claim 11 , further comprising computer-executable instructions that when executed by the computing device cause the computing device to:
receive a new interaction, wherein the new interaction is not in the corpus of interactions; and
run the large language model using the query on the received new interaction to provide automated quality management for the received new interaction.
17 . The system of claim 11 , wherein selecting the first sub-corpus comprises:
using the large language model to generate a plurality of words and phrases that are related to the query;
determining interactions in the corpus of interactions that match at least some of the plurality of words and phrases; and
selecting the determined interactions in the corpus of interactions that match at least some of the plurality of words and phrases for the first sub-corpus.
18 . The system of claim 17 , wherein selecting the second sub-corpus comprises:
determining interactions in the corpus of interactions that do not match any of the plurality of words and phrases; and
selecting the determined interactions in the corpus of interactions that do not match any of the plurality of words and phrases for the second sub-corpus.
19 . The system of claim 11 , wherein selecting the first sub-corpus comprises:
vectorizing the text query;
vectorizing each phrase of a plurality of phrases from each interaction of the corpus of interactions;
for each interaction of the corpus of interactions, determine a similarity of each vectorized phrase with the vectorized text query; and
selecting the interactions from the corpus of interactions for the first sub-corpus based on the determined similarities.
20 . The system of claim 11 , wherein selecting the first sub-corpus comprises randomly selecting a number of interactions from the corpus of interactions.