Systems and methods for composed variational natural language generation
Embodiments described herein provide a composed variational natural language generation (CLANG) model that is configured to generate training samples for few-shot intents. Specifically, the CLANG model may build connections between existing training samples of many-shot intents and new training samples of few-shot intents by modeling an intent as a combination of a domain and an action. In this way, the CLANG model transfers knowledge from existing many-shot intents to few-shot intents in natural language generation by learning how to compose utterances with many-shot intents and transferring such knowledge to few-shot intents.
1. A system for composed variational natural language generation, the system comprising:
an input interface configured to:
obtain, from a training dataset for a language model, an utterance associated with an intent of a domain and an action, and
convert the utterance and the intent into an input sequence of tokens; and
a bi-latent variational encoder configured to:
apply an attention mask that prevents a first token representing the domain and a second token representing the action from the input sequence from attending to each other,
generate a first latent variable corresponding to the first token and a second latent variable corresponding to the second token,
wherein the first latent variable and the second latent variable are disentangled from each other conditioned on the domain and the action, respectively, and
output, to a decoder, an encoded sequence of tokens including at least the first latent variable and the second latent variable.
2. The system of claim 1 , wherein the input sequence of tokens is generated by concatenating the first token representing the domain, the second token representing the action and a plurality of tokens from the utterance, separated by one or more separation tokens.
3. The system of claim 1 , wherein the bi-latent variational encoder includes:
one or more transformer layers, wherein at least one of the one or more transformer layers is configured to:
compute a triple of queries, keys and values based on an output of a previous transformer layer and a triple of a query matrix, a key matrix and a value matrix, respectively; and
compute a self-attention head based on the triple of queries, keys and values and an attention mask matrix representing the attention mask.
4. The system of claim 3 , wherein the attention mask matrix has at least one element having a value of minus infinity corresponding to a relationship between the first token and the second token thereby preventing the first token from attending to the second token.
5. The system of claim 3 , wherein the one or more transformer layers include a last transformer layer that is configured to output a first embedding and a second embedding representing that is disentangled from the second embedding.
6. The system of claim 5 , wherein the bi-latent variational encoder is further configured to generate the first latent variable corresponding to the first token and the second latent variable corresponding to the second token by:
generating a first set of probability distribution parameters based on the first embedding and pre-defined parameters;
sampling the first latent variable by a Gaussian distributed variable from a first conditional distribution of the first latent variable conditioned on the domain and the utterance, wherein the first conditional distribution is modeled based on the first set of probability distribution parameters.
7. The system of claim 1 , further comprising:
a decoder including one or more transformer layers, wherein the one or more transformer layers are configured to:
receive, from the bi-latent variational encoder, latent information representing the first latent variable and the second latent variable,
receive, from the input interface, the input sequence of tokens, and
generate a decoder input sequence of tokens by replacing a start token from the input sequence with a latent token representing the latent information.
8. The system of claim 7 , wherein the decoder is further configured to:
apply a decoder attention mask to the decoder input sequence of tokens,
wherein the decoder attention mask prevents intent tokens in the decoder input sequence of tokens from attending non-intent tokens and prevents the latent token from attending other tokens.
9. The system of claim 7 , wherein the decoder is further configured to concatenate the latent token to all other token embeddings from a last transformer layer in the one or more transformer layers, and
the decoder further comprises:
at least one fully-connected layer configured to sequentially decode the concatenated latent token and the token embeddings into tokens corresponding to the utterance by next token prediction.
10. The system of claim 7 , wherein the decoder is further configured to generate utterances for few-shot intents based on the first latent variable sampled from a first multivariate standard Gaussian distribution and the second latent variable sampled from a second multivariate standard Gaussian distribution, and
wherein the generated utterances with the few-shot intents are added to the training dataset for the language model.
11. A method for composed variational natural language generation, the method comprising:
obtaining, via an input interface, from a training dataset for a language model, an utterance associated with an intent of a domain and an action;
converting the utterance and the intent into an input sequence of tokens;
applying, by a bi-latent variation encoder, an attention mask that prevents a first token representing the domain and a second token representing the action from the input sequence from attending to each other;
generating a first latent variable corresponding to the first token and a second latent variable corresponding to the second token,
wherein the first latent variable and the second latent variable are disentangled from each other conditioned on the domain and the action, respectively; and
outputting, to a decoder, an encoded sequence of tokens including at least the first latent variable and the second latent variable.
12. The method of claim 11 , wherein the input sequence of tokens is generated by concatenating the first token representing the domain, the second token representing the action and a plurality of tokens from the utterance, separated by one or more separation tokens.
13. The method of claim 11 , further comprising:
computing, by one or more transformer layers at the bi-latent variation encoder, a triple of queries, keys and values based on an output of a previous transformer layer and a triple of a query matrix, a key matrix and a value matrix, respectively; and
computing a self-attention head based on the triple of queries, keys and values and an attention mask matrix representing the attention mask.
14. The method of claim 13 , wherein the attention mask matrix has at least one element having a value of minus infinity corresponding to a relationship between the first token and the second token thereby preventing the first token from attending to the second token.
15. The method of claim 13 , wherein the one or more transformer layers include a last transformer layer that is configured to output a first embedding and a second embedding representing that is disentangled from the second embedding.
16. The method of claim 15 , further comprising:
generating the first latent variable corresponding to the first token and the second latent variable corresponding to the second token by:
generating a first set of probability distribution parameters based on the first embedding and pre-defined parameters;
sampling the first latent variable by a Gaussian distributed variable from a first conditional distribution of the first latent variable conditioned on the domain and the utterance, wherein the first conditional distribution is modeled based on the first set of probability distribution parameters.
17. The method of claim 11 , further comprising:
receiving, at a decoder from the bi-latent variational encoder, latent information representing the first latent variable and the second latent variable,
receiving, from the input interface, the input sequence of tokens, and
generating a decoder input sequence of tokens by replacing a start token from the input sequence with a latent token representing the latent information.
18. The method of claim 17 , further comprising:
applying a decoder attention mask to the decoder input sequence of tokens,
wherein the decoder attention mask prevents intent tokens in the decoder input sequence of tokens from attending non-intent tokens and prevents the latent token from attending other tokens.
19. The method of claim 18 , further comprising:
concatenating the latent token to all other token embeddings from a last transformer layer in the one or more transformer layers, and
sequentially decoding, by at least one fully-connected layer, the concatenated latent token and the token embeddings into tokens corresponding to the utterance by next token prediction.
20. The method of claim 11 , further comprising:
generating utterances for few-shot intents based on the first latent variable sampled from a first multivariate standard Gaussian distribution and the second latent variable sampled from a second multivariate standard Gaussian distribution, and
wherein the generated utterances with the few-shot intents are added to the training dataset for the language model.