Systems and methods for composed variational natural language generation
Embodiments described herein provide a composed variational natural language generation (CLANG) model that is configured to generate training samples for few-shot intents. Specifically, the CLANG model may build connections between existing training samples of many-shot intents and new training samples of few-shot intents by modeling an intent as a combination of a domain and an action. In this way, the CLANG model transfers knowledge from existing many-shot intents to few-shot intents in natural language generation by learning how to compose utterances with many-shot intents and transferring such knowledge to few-shot intents.
1. A system for training a composed variational natural language generator implemented on a neural network, the system comprising:
an input interface configured to receive a training sequence of tokens including a first intent token and a second intent token;
a memory storing the composed variational natural language generator including an encoder and a decoder, and a plurality of processor-executable instructions; and
one or more processors executing the plurality-executable instructions to:
encode, by the encoder, the training sequence of tokens into an encoded sequence including a first latent variable corresponding to the first intent token and a second latent variable corresponding to the second intent token, wherein the first intent token and the second intent token are prevented from attending to each other during encoding;
compute an encoder loss based on a first conditional distribution of the first latent variable conditioned on the first intent token and a second conditional distribution of the second latent variable conditioned on the second intent token;
generate, by the decoder and from the encoded sequence, a reconstructed sequence of tokens;
compute a reconstruction loss based on a third conditional distribution of the reconstructed sequence of tokens conditioned on the first intent token, the second intent token, the first latent variable and the second latent variable;
train the composed variational natural language generator by updating parameters of the encoder and the decoder based at least in part on the encoder loss and the reconstruction loss; and
generate, by the trained composed variational natural language generator, a response output in response to an input of a testing utterance.
2. The system of claim 1 , wherein the encoder loss is computed based on a first KL-loss between the first conditional distribution of the first latent variable conditioned on the first intent token and a fourth conditional distribution of the first latent variable conditioned on both the first intent token and the training sequence of tokens, and a second KL-loss between the second conditional distribution of the second latent variable conditioned on the second intent token and a fifth conditional distribution of the second latent variable conditioned on both the second intent token and the training sequence of tokens.
3. The system of claim 2 , wherein the reconstruction loss is computed as an expectation of a logarithm of the third conditional distribution, wherein the expectation is conditioned on the fourth conditional distribution and the fifth conditional distribution.
4. The system of claim 1 , wherein input interface is further configured to:
receive an in-class training sample and an out-of-class training sample for the first intent token and the second intent token.
5. The system of claim 4 , wherein the in-class training sample corresponds to the first intent token and the second intent token, and the out-of-class training sample is generated by selecting, from a set of training samples, a training sample that is similar to the in-class training sample and corresponds to an intent other than the first intent token or the second intent token.
6. The system of claim 5 , wherein the selected training sample has a similarity with the in-class training sample based on a first number of shared unigrams between the selected training sample and the in-class training sample, a second number of shared bi-grams between the selected training sample and the in-class training sample, and a third number of shared uni-grams between names of intents corresponding to the selected training sample and the in-class training sample.
7. The system of claim 4 , wherein the encoder is configured to:
encode the in-class training sample with the first intent token and the second intent token into a first encoded sequence, and the out-of-class training sample with the first intent token and the second intent token into a second encoded sequence, respectively; and
wherein the decoder is configured to:
generate a reconstructed in-class sample from the first encoded sequence and a reconstructed out-of-class sample from the second encoded sequence, respectively.
8. The system of claim 7 , wherein the decoder is configured to sequentially generate each token in the reconstructed in-class sample or in the reconstructed out-of-class sample, and to sequentially compute a first conditional probability of the reconstructed in-class sample and a second conditional probability of a reconstructed out-of-class sample, both conditioned on the first intent token and the second token.
9. The system of claim 8 , wherein the decoder is further configured to:
compute a contrastive loss by constraining a logarithm of the first conditional probability to be higher than that of the second conditional probability for at least a pre-defined margin.
10. The system of claim 9 , wherein the encoder and the decoder are jointly trained by any combination of the encoder loss, the decoder loss and the contrastive loss.
11. A method for training a composed variational natural language generator implemented on a neural network and comprising an encoder and a decoder, the method comprising:
receiving, via an input interface, a training sequence of tokens including a first intent token and a second intent token;
encoding, by the encoder, the training sequence into an encoded sequence including a first latent variable corresponding to the first intent token and a second latent variable corresponding to the second intent token, wherein the first intent token and the second intent token are prevented from attending to each other during encoding;
computing an encoder loss based on a first conditional distribution of the first latent variable conditioned on the first intent token and a second conditional distribution of the second latent variable conditioned on the second intent token;
generating, by the decoder and from the encoded sequence, a reconstructed sequence of tokens;
computing a reconstruction loss based on a third conditional distribution of the reconstructed sequence of tokens conditioned on the first intent token, the second intent token, the first latent variable and the second latent variable;
training the composed variational natural language generator by updating parameters of the encoder and the decoder based at least in part on the encoder loss and the reconstruction loss; and
generating, by the trained composed variational natural language generator, a response output in response to an input of a testing utterance.
12. The method of claim 11 , wherein the encoder loss is computed based on a first KL-loss between the first conditional distribution of the first latent variable conditioned on the first intent token and a fourth conditional distribution of the first latent variable conditioned on both the first intent token and the training sequence of tokens, and a second KL-loss between the second conditional distribution of the second latent variable conditioned on the second intent token and a fifth conditional distribution of the second latent variable conditioned on both the second intent token and the training sequence of tokens.
13. The method of claim 12 , wherein the reconstruction loss is computed as an expectation of a logarithm of the third conditional distribution, wherein the expectation is conditioned on the fourth conditional distribution and the fifth conditional distribution.
14. The method of claim 11 , further comprising:
receiving an in-class training sample and an out-of-class training sample for the first intent token and the second intent token.
15. The method of claim 14 , wherein the in-class training sample corresponds to the first intent token and the second intent token, and the out-of-class training sample is generated by selecting, from a set of training samples, a training sample that is similar to the in-class training sample and corresponds to an intent other than the first intent token or the second intent token.
16. The method of claim 15 , wherein the selected training sample has a similarity with the in-class training sample based on a first number of shared unigrams between the selected training sample and the in-class training sample, a second number of shared bi-grams between the selected training sample and the in-class training sample, and a third number of shared uni-grams between names of intents corresponding to the selected training sample and the in-class training sample.
17. The method of claim 14 , further comprising:
encoding the in-class training sample with the first intent token and the second intent token into a first encoded sequence, and the out-of-class training sample with the first intent token and the second intent token into a second encoded sequence, respectively; and
generating a reconstructed in-class sample from the first encoded sequence and a reconstructed out-of-class sample from the second encoded sequence, respectively.
18. The method of claim 17 , further comprising:
sequentially generating each token in the reconstructed in-class sample or in the reconstructed out-of-class sample; and
sequentially computing a first conditional probability of the reconstructed in-class sample and a second conditional probability of a reconstructed out-of-class sample, both conditioned on the first intent token and the second token.
19. The method of claim 18 , further comprising:
computing a contrastive loss by constraining a logarithm of the first conditional probability to be higher than that of the second conditional probability for at least a pre-defined margin.
20. The method of claim 19 , further comprising:
updating the composed variational natural language generator based on any combination of the encoder loss, the decoder loss and the contrastive loss.