Method and apparatus for consistency detection and resolution in automatic dialogue systems
A method for performing consistency detection in a dialogue system includes training a model based on an annotated dataset for consistency detection to generate a trained model. The method further includes receiving a plurality of utterances from one or more users. The method includes determining, using the trained model, whether the plurality of utterances include an inconsistency. The method includes, based on the determination that the plurality of utterances include the inconsistency, generating, using the trained model, one or more clarifying responses.
1 . A method performed by at least one processor retrieving one or more instructions from memory for performing consistency detection in a dialogue system, the method comprising:
training a model based on an annotated dataset for consistency detection to generate a trained model, wherein the trained model comprises (i) an inconsistency checker model that is trained based on the annotated dataset to detect inconsistencies in a dialogue, and (ii) a resolver model that is trained based on the annotated data set to output the one or more clarifying responses, wherein each of the inconsistency checker model and the resolver model comprises an encoder-decoder structure that inputs the dialogue and auto-aggressively outputs at least one clarification response;
receiving a plurality of utterances corresponding to a user-to-chatbot conversation comprising at least one user utterance and at least one chatbot response;
determining, using the trained model, whether the plurality of utterances include an inconsistency; and
based on the determination that the plurality of utterances include the inconsistency, generating, using the trained model, one or more clarifying responses.
2 . The method of claim 1 , wherein the annotated dataset comprises a training utterance and an annotated utterance that is inconsistent with the training utterance.
3 . The method of claim 2 , wherein the annotate dataset comprises an annotated explanation of the inconsistency prepared in accordance with a template.
4 . The method of claim 2 , wherein the annotated dataset includes an annotated clarifying response.
5 . The method of claim 1 , wherein the inconsistency is determined by performing a pair check between a first utterance from the plurality of utterances and a second utterance from the plurality of utterances.
6 . The method of claim 1 , wherein the inconsistency is determined by performing a diagonal check between a current utterance from the plurality of utterances and two or more utterances from the plurality of utterances that are prior to the current utterance.
7 . An apparatus method for performing consistency detection in a dialogue system, the method comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code including:
training code configured to cause the at least one processor to train a model based on an annotated dataset for consistency detection to generate a trained model, wherein the trained model comprises (i) an inconsistency checker model that is trained based on the annotated dataset to detect inconsistencies in a dialogue, and (ii) a resolver model that is trained based on the annotated data set to output the one or more clarifying responses, wherein each of the inconsistency checker model and the resolver model comprises an encoder-decoder structure that inputs the dialogue and auto-aggressively outputs at least one clarification response,
receiving code configured to cause the at least one processor to receive a plurality of utterances corresponding to a user-to-chatbot conversation comprising at least one user utterance and at least one chatbot response,
determining code configured to cause the at least one processor to determine, using the trained model, whether the plurality of utterances include an inconsistency, and
generating code configured to cause the at least one processor to, based on the determination that the plurality of utterances include the inconsistency, generate using the trained model, one or more clarifying responses.
8 . The apparatus of claim 7 , wherein the annotated dataset comprises a training utterance and an annotated utterance that is inconsistent with the training utterance.
9 . The apparatus of claim 8 , wherein the annotate dataset comprises an annotated explanation of the inconsistency prepared in accordance with a template.
10 . The apparatus of claim 8 , wherein the annotated dataset includes an annotated clarifying response.
11 . The apparatus of claim 7 , wherein the inconsistency is determined by performing a pair check between a first utterance from the plurality of utterances and a second utterance from the plurality of utterances.
12 . The apparatus of claim 7 , wherein the inconsistency is determined by performing a diagonal check between a current utterance from the plurality of utterances and two or more utterances from the plurality of utterances that are prior to the current utterance.
13 . A non-transitory computer readable medium having instructions stored therein, which when executed by a processor cause the processor to execute a method for performing consistency detection in a dialogue system, the method comprising:
training a model based on an annotated dataset for consistency detection to generate a trained model, wherein the trained model comprises (i) an inconsistency checker model that is trained based on the annotated dataset to detect inconsistencies in a dialogue, and (ii) a resolver model that is trained based on the annotated data set to output the one or more clarifying responses, wherein each of the inconsistency checker model and the resolver model comprises an encoder-decoder structure that inputs the dialogue and auto-aggressively outputs at least one clarification response;
receiving a plurality of utterances corresponding to a user-to-chatbot conversation comprising at least one user utterance and at least one chatbot response;
determining, using the trained model, whether the plurality of utterances include an inconsistency; and
based on the determination that the plurality of utterances include the inconsistency, generating, using the trained model, one or more clarifying responses.
14 . The non-transitory computer readable medium of claim 13 , wherein the annotated dataset comprises a training utterance and an annotated utterance that is inconsistent with the training utterance.
15 . The non-transitory computer readable medium of claim 14 , wherein the annotate dataset comprises an annotated explanation of the inconsistency prepared in accordance with a template.
16 . The non-transitory computer readable medium of claim 14 , wherein the annotated dataset includes an annotated clarifying response.
17 . The non-transitory computer readable medium of claim 13 , wherein the inconsistency is determined by performing a pair check between a first utterance from the plurality of utterances and a second utterance from the plurality of utterances.
18 . The non-transitory computer readable medium of claim 13 , wherein the inconsistency is determined by performing a diagonal check between a current utterance from the plurality of utterances and two or more utterances from the plurality of utterances that are prior to the current utterance.