Learning dialogue state tracking with limited labeled data
Embodiments described in this disclosure illustrate the use of self-/semi supervised approaches for label-efficient DST in task-oriented dialogue systems. Conversational behavior is modeled by next response generation and turn utterance generation tasks. Prediction consistency is strengthened by augmenting data with stochastic word dropout and label guessing. Experimental results show that by exploiting self-supervision the joint goal accuracy can be boosted with limited labeled data.
1. A method for boosting prediction consistency in a dialogue state tracking (DST) module, the method comprising:
receiving, from a database, a first dataset of dialogue data;
generating, via the DST module, a first predicted attention distribution and a first predicted slot gate distribution from the first dataset of dialogue data;
generating multiple datasets of masked dialogue data by masking one or more words in the first dataset of dialogue data over multiple iterations;
generating, via the DST module, a set of predicted attention distributions and predicted slot gate distributions from the masked dialogue data;
computing an average predicted attention distribution and an average predicted slot gate distribution from the first dataset of dialogue data and the multiple datasets of masked dialogue data;
computing a consistency loss metric based on a first error metric between the average predicted attention distribution and the first predicted attention distribution obtained from unmasked dialogue data, and a second error metric between the average predicted slot gate distribution and the first predicted slot gate distribution obtained from unmasked dialogue data; and
incorporating the consistency loss metric into a loss metric for updating the DST module.
2. The method of claim 1 , wherein the one or more words are randomly selected from the first dataset of dialogue data.
3. The method of claim 1 , wherein the multiple datasets of masked dialogue data are generated by masking different words from the first dataset of dialogue data at different iterations.
4. The method of claim 1 , wherein the multiple datasets of masked dialogue data are generated by masking a same word from different places in the first dataset of dialogue data at different iterations.
5. The method of claim 1 , wherein the average predicted attention distribution is generated for each domain-slot pair by:
generating, using the DST module, a respective conditional attention distribution conditioned on DST module parameters and each of the multiple datasets of masked dialogue data or the first dataset of dialogue data; and
computing the average predicted attention distribution for each domain-slot pair by taking an average of the generated conditional attention distributions.
6. The method of claim 5 , wherein the average predicted slot gate distribution is generated for each domain-slot pair by:
generating, using the DST module, a respective conditional slot gate distribution conditioned on DST module parameters and each of the multiple datasets of masked dialogue data or the first dataset of dialogue data; and
computing the average predicted slot gate distribution for each domain-slot pair by taking an average of the generated conditional slot gate distributions.
7. The method of claim 6 , further comprising:
adjusting the average predicted slot gate distribution by applying a sharpening function that reduces an entropy corresponding to the average predicted slot gate distribution.
8. The method of claim 1 , wherein the first error metric or the second error metric is obtained by a mean-squared operation.
9. The method of claim 1 , wherein the consistency loss metric is computed by summing all error metrics corresponding to all domain-slot pairs.
10. The method of claim 1 , wherein the first dataset of dialogue data includes unlabeled dialogue data, and the method further comprises:
generating an unsupervised loss metric for the DST module using the unlabeled dialogue data; and
using a weighted sum of the unsupervised loss metric and the consistency loss metric for updating the DST module.
11. A system for boosting prediction consistency in a dialogue state tracking (DST) module, the system comprising:
a memory storing instructions for boosting prediction consistency in the DST module; and
a processor configured to read and execute the instructions from the memory to:
receive, from a database, a first dataset of dialogue data;
generate, via the DST module, a first predicted attention distribution and a first predicted slot gate distribution from the first dataset of dialogue data;
generate multiple datasets of masked dialogue data by masking one or more words in the first dataset of dialogue data over multiple iterations;
generate, via the DST module, a set of predicted attention distributions and predicted slot gate distributions from the masked dialogue data;
compute an average predicted attention distribution and an average predicted slot gate distribution from the first dataset of dialogue data and the multiple datasets of masked dialogue data;
compute a consistency loss metric based on a first error metric between the average predicted attention distribution and the first predicted attention distribution obtained from unmasked dialogue data, and a second error metric between the average predicted slot gate distribution and the first predicted slot gate distribution obtained from unmasked dialogue data; and
incorporate the consistency loss metric into a loss metric for updating the DST module.
12. The system of claim 11 , wherein the one or more words are randomly selected from the first dataset of dialogue data.
13. The system of claim 11 , wherein the multiple datasets of masked dialogue data are generated by masking different words from the first dataset of dialogue data at different iterations.
14. The system of claim 11 , wherein the multiple datasets of masked dialogue data are generated by masking a same word from different places in the first dataset of dialogue data at different iterations.
15. The system of claim 11 , wherein the average predicted attention distribution is generated for each domain-slot pair by:
generating, using the DST module, a respective conditional attention distribution conditioned on DST module parameters and each of the multiple datasets of masked dialogue data or the first dataset of dialogue data; and
computing the average predicted attention distribution for each domain-slot pair by taking an average of the generated conditional attention distributions.
16. The system of claim 15 , wherein the average predicted slot gate distribution is generated for each domain-slot pair by:
generating, using the DST module, a respective conditional slot gate distribution conditioned on DST module parameters and each of the multiple datasets of masked dialogue data or the first dataset of dialogue data; and
computing the average predicted slot gate distribution for each domain-slot pair by taking an average of the generated conditional slot gate distributions.
17. The system of claim 16 , wherein the processor is further configured to execute instructions from the memory to:
adjust the average predicted slot gate distribution by applying a sharpening function that reduces an entropy corresponding to the average predicted slot gate distribution.
18. The system of claim 11 , wherein the first error metric or the second error metric is obtained by a mean-squared operation.
19. The system of claim 11 , wherein the consistency loss metric is computed by summing all error metrics corresponding to all domain-slot pairs.
20. A non-transitory processor-readable storage medium storing processor-executable instructions for boosting prediction consistency in a dialogue state tracking (DST) module, the instructions being executed by a processor to:
receive, from a database, a first dataset of dialogue data;
generate, via the DST module, a first predicted attention distribution and a first predicted slot gate distribution from the first dataset of dialogue data;
generate multiple datasets of masked dialogue data by masking one or more words in the first dataset of dialogue data over multiple iterations;
generate, via the DST module, a set of predicted attention distributions and predicted slot gate distributions from the masked dialogue data;
compute an average predicted attention distribution and an average predicted slot gate distribution from the first dataset of dialogue data and the multiple datasets of masked dialogue data;
compute a consistency loss metric based on a first error metric between the average predicted attention distribution and the first predicted attention distribution obtained from unmasked dialogue data, and a second error metric between the average predicted slot gate distribution and the first predicted slot gate distribution obtained from unmasked dialogue data; and
incorporate the consistency loss metric into a loss metric for updating the DST module.