Method and system for predicting understanding errors in a task classification system
View Patent ↗A method and system for monitoring an automated dialog system for the automatic recognition of language understanding errors based on a user's input communications in a task classification system. If the user's input communication cannot be understood and a task classification decision cannot be made, then further dialog may be conducted with the user if a probability of understanding the user's input communication exceeds a first threshold. Otherwise, the user may be directed to a human for assistance. In another possible embodiment, the method operates as above except that if the probability exceeds a second threshold, then further dialog may be conducted with the user using the current dialog strategy. However, if the probability falls between a first threshold and a second threshold, the dialog strategy may be adapted in order to improve the chances of conducting a successful dialog with the user.
1. A language understanding monitoring system that operates in a task classification system, comprising
a task classification processor that is adapted to determine whether a task classification decision can be made based on an understanding of a user's input communication;
a dialog manager that is adapted to conduct a dialog with the user; and
a language understanding monitor that is adapted to determine whether a probability of understanding the users input communication exceeds a first threshold in the cases where a task classification decision cannot be made and, if the first threshold is exceeded, to prompt the dialog manager to conduct further dialog with the user,
wherein the language understanding monitor is further adapted to determine whether a probability of understanding the user's input communication exceeds a second threshold, the second threshold being greater than the first threshold, and if the second threshold is exceeded, to prompt the dialog manager to conduct further dialog with the user using a current dialog strategy, and
wherein at least one of said task classification processor, said dialog manager and said language understanding monitor performs that which it is adapted to do using a programmed processor.
2. The system of claim 1 wherein the language understanding monitor is further adapted to prompt the dialog manager to route the user to a human for assistance if the first threshold is not exceeded.
3. The system of claim 1 wherein the language understanding monitor is further adapted to prompt the dialog manager to conduct further dialog with the user using an adapted dialog strategy if the second threshold is not exceeded.
4. The system of claim 3 wherein the adapted dialog strategy includes one of prompting the user with choices and prompting the user to confirm the recognition and understanding data.
5. The system of claim 1 wherein the language understanding monitor is further adapted to determine the probability using recognition data provided by a recognizer and understanding data provided by a language understanding unit, and the recognition and understanding data is derived from the user's input communication.
6. The system of claim 1 further comprising a training database for storing training data for language understanding, wherein the language understanding monitor is further adapted to determine the probability using the training data stored in the training database, the training data including at least one of classification models and extracted features.
7. The system of claim 6 wherein the extracted features are derived from recognition, understanding and dialog data.
8. The system of claim 1 further comprising
a dialog history database that is adapted to store a first dialog exchange, wherein the first dialog exchange includes a first automated dialog output and the user's first input communication, and the further dialog conducted with the user results in a second dialog exchange, the second dialog exchange including a second dialog output and the user's second input communication, and the language understanding monitor being further adapted to determine whether the probability of understanding exceeds the first threshold using the first dialog exchange and the second dialog exchange.
9. The system of claim 8 wherein the language understanding monitor is further adapted to determine whether the probability of understanding exceeds the first threshold using each of the dialog exchanges conducted.
10. The system of claim 1 wherein the task classification processor is further adapted to a) classify the task as one of a set of task objectives and to route the user's input communication in accordance with the classified task objective, if b) the task classification processor can make a task classification decision.
11. The system of claim 1 further comprising
a recognizer that is adapted to recognize the user's input communication; and
a language understanding unit that is adapted to apply a confidence function to the recognized portions of the user's input communication and to provide an input to the language understanding monitor.
12. The system of claim 11 wherein the language understanding monitor is further adapted to determine the probability using recognition data from the recognizer and understanding data from the language understanding unit derived from the user's input communication.