Identifying high effort statements for call center summaries
Disclosed herein are system, method, and computer program product embodiments for machine learning systems to process incoming call-center calls to provide communication summaries that capture effort levels of statements made during interactive communications. For a given call, the system receives a transcript as the input and generates a textual summary as the output. In order to improve a call summary and customize a summarization task to a call center domain, the technology disclosed herein may employ a classifier that predicts an effort level and attention score for individual utterances within a call transcript, ranks the attention scores and uses selected ones of the ranked utterances in the summary.
1. A system for ranking utterances in a natural language processing environment, the system comprising:
a speech recognizer configured to:
receive an interactive communication between a first participant and a second participant;
extract individual utterances of the first participant; and
convert the individual utterances to a textual transcript of individual utterances of the first participant;
a machine learning engine configured to:
evaluate the textual transcript of individual utterances of the first participant, using a customer effort classifier, to determine a binary effort level label for a plurality of the individual utterances of the first participant;
evaluate the textual transcript of individual utterances of the first participant, using the customer effort classifier, to generate attention scores for the plurality of the individual utterances of the first participant, wherein the attention scores are based at least partially on a calculated importance of the individual utterances relative to the plurality of the individual utterances of the first participant;
and
an interactive communication summarizer configured to:
rank the plurality of the individual utterances of the first participant based on the binary effort level label and the attention scores;
select, based on the ranking, one or more utterances of the plurality of the individual utterances of the first participant; and
generate a summary of the interactive communication with the selected one or more utterances.
2. The system of claim 1 , wherein the customer effort classifier is configured to classify the plurality of the individual utterances of the first participant based on a deep learning customer effort predictive model.
3. The system of claim 2 , wherein the first participant is a customer and the deep learning customer effort predictive model is trained, by the machine learning engine, based on training parameters comprising a plurality of customer satisfaction outcomes.
4. The system of claim 3 , wherein the training parameters further comprise a recorded customer level of effort by the second participant.
5. The system of claim 3 , wherein the training parameters further comprise a recorded customer level of effort by the first participant.
6. The system of claim 1 , wherein the calculated importance is based on the machine learning engine assigning a weighted importance value for the plurality of the individual utterances of the first participant.
7. The system of claim 1 , wherein the binary effort level label reflects a classification of a high effort level or a low effort level.
8. The system of claim 7 , wherein the binary effort level label is classified as the high effort level based on an aggregation of the attention scores reaching a threshold.
9. The system of claim 1 , wherein the system further comprises a feedback system configured to:
assess a quality of the generated summary and feed a corresponding assessment to the machine learning engine.
10. A computer implemented method for ranking utterances in a natural language processing environment, comprising:
receiving, by a speech classifier, an interactive communication between a first participant and a second participant;
extracting, by the speech classifier, individual utterances of the first participant;
converting, by the speech classifier, the individual utterances to a textual transcript of individual utterances of the first participant;
evaluating, by a machine learning customer effort classification model, the textual transcript of individual utterances of the first participant to determine a binary effort level label for a plurality of the individual utterances by the first participant;
evaluating, by the machine learning customer effort classification model, the textual transcript of individual utterances of the first participant to generate attention scores for the plurality of the individual utterances by the first participant, wherein the attention scores are based at least partially on a calculated importance of the individual utterances relative to the plurality of the individual utterances;
and
ranking, by a summarization model, the plurality of the individual utterances based on the binary effort level label and the attention scores;
selecting, by the summarization model and based on the ranking, one or more utterances of the plurality of the individual utterances from the interactive communication; and
generating, by the summarization model, a summary of the interactive communication with at least the one or more utterances.
11. The computer implemented method of claim 10 , further comprising selecting the one or more utterances by selecting one or more of highest ranked ones of the plurality of the individual utterances.
12. The computer implemented method of claim 10 , further comprising training the machine learning customer effort classification model based on a plurality of customer satisfaction outcomes.
13. The computer implemented method of claim 12 , wherein the plurality of customer satisfaction outcomes are based on a customer level of effort recorded by the second participant.
14. The computer implemented method of claim 10 , wherein the calculated importance is based on the machine learning customer effort classification model assigning a weighted importance value for the plurality of the individual utterances.
15. The computer implemented method of claim 10 , further comprising that the binary effort level label is a high effort level or a low effort level.
16. The computer implemented method of claim 10 , further comprising receiving, by the machine learning customer effort classification model, a quality of the generated summary as training feedback.
17. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform natural language processing operations comprising:
receiving, by a speech classifier, an interactive communication between a first participant and a second participant;
extracting, by the speech classifier, individual utterances of the first participant; and
converting, by the speech classifier, the individual utterances to a textual transcript of individual utterances of the first participant;
evaluating, by a machine learning customer effort classification model, the textual transcript of individual utterances of the first participant to determine a binary effort level label for a plurality of the individual utterances by the first participant;
evaluating, by the machine learning customer effort classification model, the textual transcript of individual utterances of the first participant to generate attention scores for the plurality of the individual utterances by the first participant, wherein the attention scores are based at least partially on a calculated importance of the individual utterances relative to the plurality of the individual utterances;
and
ranking, by a summarization model, the plurality of the individual utterances based on the binary effort level label and the attention scores;
selecting, by the summarization model and based on the ranking, one or more utterances of the plurality of the individual utterances from the interactive communication; and
generating, by the summarization model, a summary of the interactive communication with the selected one or more utterances.
18. The non-transitory computer-readable device of claim 17 , further configured to perform operations comprising:
training the machine learning customer effort classification model based on a plurality of customer satisfaction outcomes.
19. The non-transitory computer-readable device of claim 18 , wherein the plurality of customer satisfaction outcomes are based on a customer level of effort recorded by the second participant.
20. The non-transitory computer-readable device of claim 17 , further configured to perform operations comprising:
receiving an assessment of a quality of the generated summary as training data and feeding the training data to the machine learning customer effort classification model.