System and method for robust evaluation of the user experience in automated spoken dialog systems
View Patent ↗A single, subjective numerical rating to evaluate the performance of a telephone-based spoken dialog system. This CE rating is provided by expert human listeners who have knowledge of the design of the dialog system. Different human raters can be trained to achieve a satisfactory level of agreement. Furthermore, a classifier trained on ratings by human experts can reproduce the human ratings with the same degree of consistency. More calls can be given a CE rating than would be possible with limited human resources. More information can be provided about individual calls, e.g., to help decide between two disparate ratings by different human experts.
1. An interactive spoken dialog system, comprising:
a computer including a processor, and non-transitory memory, including:
a data store for storing recordings of interactions on the spoken dialog system;
a data store for feature data defining a number of features, wherein the features include a set of features selected from the group of:
the classification status of a call,
the number of speech recognition errors during a call,
the number of operator requests from a caller, and
the exit status of the call; and
an automated rater including a machine-learning algorithm configured to automatically estimate a user experience score for any set of interactions on the spoken dialog system.
2. The system of claim 1 wherein system further comprises
a data store for storing logging data, wherein the logging data includes a speech log of a spoken dialog system, whereby the feature data is extracted from said logging data.
3. The system of claim 1 wherein the automated rater automatically estimates the user experience score for any set of interactions on the interactive spoken dialog system for the same features.
4. The system of claim 1 wherein the system is configured to use the rater's automated estimate in conjunction with a human evaluator estimates to score the set of interaction recordings to obtain an overall user experience score.
5. The system of claim 1 wherein the machine learning algorithm includes a classifier.
6. The system of claim 5 wherein the classifier includes a classifier selected from the group of:
a decision tree;
a propositional rule learner; and
linear regression.
7. The system of claim 4 , wherein the system configured to obtain human evaluator estimates in the manner comprising:
selecting a set of the recorded interactions for the spoken dialog system;
selecting a number of the evaluators to analyze the set;
having each evaluator score the recordings;
accepting a score of the interactions that do not show any scoring difference across different evaluators;
assigning an average score to the interactions that have an absolute predetermined score difference among the evaluators;
performing an adjudication process for all the interactions that have a score difference among the evaluators that is larger than the predetermined score difference to obtain an reevaluated score; and
averaging the assigned average score and the reevaluated score to obtain an estimated value of the user experience.
8. The system of claim 7 wherein the system configured to obtain human evaluator estimates in the manner further comprising:
having the evaluators reevaluate each interaction;
having evaluators resubmit a reevaluated score; and
repeating the reevaluation and resubmission until the difference in scores for each interaction is no greater than the predetermined score difference.
9. A method to estimate a user experience score for an interactive spoken dialog system, comprising:
selecting a number of evaluators to score a number of interactions in an interactive spoken dialog system;
having each evaluator individually score the interaction recordings to evaluate the user experience of the interactive spoken dialog system; and
obtaining a value that reduces the variability in the individual scoring;
wherein the interactive spoken dialog system comprises a computer including a processor, and non-transitory memory, including:
a data store for storing recordings of interactions on the spoken dialog system;
a data store for feature data defining a number of features, wherein the features include a set of features selected from the group of:
the classification status of a call,
the number of speech recognition errors during a call,
the number of operator requests from a caller, and
the exit status of the call; and
an automated rater including a machine-learning algorithm configured to automatically estimate a user experience score for any set of interactions on the spoken dialog system, wherein the system is configured to use the rater's automated estimate in conjunction with a human evaluator estimates to score the set of interaction recordings to obtain an overall user experience score.
10. A method to evaluate user experience in a spoken dialog system comprising:
selecting a set of recorded interactions for an interactive spoken dialog system, whereby the set represents a corpus of interactions handled by the spoken dialog interaction system;
selecting a number of evaluators to analyze the set;
having each evaluator score the recordings;
accepting a score of the interactions that do not show any scoring difference across different evaluators;
assigning an average score to the interactions that have an absolute predetermined score difference among the evaluators;
performing an adjudication process for all the interactions that have a score difference among the evaluators that is larger than the predetermined score difference to obtain an reevaluated score; and
averaging the assigned average score and the reevaluated score to obtain an estimated value of the user experience,
wherein the interactive spoken dialog system comprises a computer including a processor, and non-transitory memory, including:
a data store for storing recordings of interactions on the spoken dialog system; and
an automated rater including a machine-learning algorithm configured to automatically estimate a user experience score for any set of interactions on the spoken dialog system.
11. The method of claim 10 wherein the adjudication process comprises:
having the evaluators reevaluate each interaction;
having evaluators resubmit a reevaluated score; and
repeating the reevaluation and resubmission until the difference in scores for each interaction is no greater than the predetermined score difference.
12. The method of claim 11 wherein the evaluators reevaluate each interaction after communication amongst the evaluators about the evaluators' initial scoring.
13. The method of claim 10 wherein the predetermined score difference is 1 point.
14. The method of claim 10 wherein assigning the average score is computed as the arithmetic average among all scores assigned by each evaluators.
15. A method to automatically estimate the quality of the user experience for a spoken dialog system comprising wherein the interactive spoken dialog system comprises a computer including a processor, and non-transitory memory, including a data store for storing recordings of interactions on the spoken dialog system; and an automated rater including a machine-learning algorithm configured to automatically estimate a user experience score for any set of interactions on the spoken dialog system, the method comprising:
creating a pool of sample training interactions for the spoken dialog system;
defining a number of features for each interaction in the pool of sample training interactions, wherein the features can be derived and measured from information associated with each interaction;
associating the features of each training interaction with a value of a user experience score produced by a plurality of evaluators;
training a machine learning algorithm using features and the scores of each interaction, wherein the algorithm is trained to predict the score for the features; and
using the trained machine-learning algorithm to automatically estimate the user experience score for any set of interactions on an interactive voice response system;
obtaining human evaluator estimates by the method comprising
selecting a set of recorded interactions for an interactive spoken dialog system, whereby the set represents a corpus of interactions handled by the spoken dialog interaction system,
selecting a number of evaluators to analyze the set,
having each evaluator score the recordings,
accepting a score of the interactions that do not show any scoring difference across different evaluators,
accepting an average score to the interactions that have an absolute predetermined score difference among the evaluators,
performing an adjudication process for all the interactions the have a score difference among the evaluators that is larger than the predetermined score difference to obtain an reevaluated score, and
averaging the assigned average score and the reevaluated score to obtain an estimated value of the user experience; and
using the automated estimate in conjunction with the human evaluator estimates to score a set of interaction recordings to obtain an overall user experience score.
16. The method of claim 15 wherein the machine learning algorithm includes a classifier.
17. The method of claim 16 wherein the machine learning algorithm is selected from the group of:
a decision tree;
a propositional rule learner; and
linear regression.
18. The method of claim 15 wherein the features include measures extracted from a speech log of a spoken dialog system.
19. The method of claim 15 wherein the features include a set of features selected from the group of:
the classification status of a call,
the number of speech recognition errors during a call,
the number of operator requests from a caller, and
the exit status of the call.
20. The method of claim 15 wherein the method includes:
automatically estimating the user experience score for any set of interactions on a different spoken dialog system, or
automatically estimating the user experience score for any set of interactions on the same spoken dialog system.
21. The method of claim 15 wherein the adjudication process comprises:
having the evaluators reevaluate each interaction;
having evaluators resubmit a reevaluated score; and
repeating the reevaluation and resubmission until the difference in scores for each interaction is no greater than the predetermined score difference.