Contextual text generation for question answering and text summarization with supervised representation disentanglement and mutual information minimization
Methods and systems for disentangled data generation include accessing a dataset including pairs, each formed from a given input text structure and a given style label for the input text structures. An encoder is trained to disentangle a sequential text input into disentangled representations, including a content embedding and a style embedding, based on a subset of the dataset, using an objective function that includes a regularization term that minimizes mutual information between the content embedding and the style embedding. A generator is trained to generate a text output that includes content from the style embedding, expressed in a style other than that represented by the style embedding of the text input.
1. A computer-implemented method for disentangled data generation, comprising:
accessing a dataset including a plurality of pairs, each formed from a given one of a plurality of input text structures and a given one of a plurality of style labels for the plurality of input text structures;
training an encoder neural network to disentangle a sequential text input into disentangled representations, including a content embedding and a style embedding, based on a subset of the dataset, using an objective function that includes a regularization term that minimizes mutual information between the content embedding and the style embedding, wherein the objective function is:
VAE +λ reg
wherein λ is a hyperparameter reweighting a regularization reg and a variational autoencoder objective VAE , where reg is expressed as Dis +MI(s; c), including a disentanglement loss Dis and a mutual information term MI(s; c) based on a style embedding s and a content embedding c; and
training a generator neural network to generate a text output that includes content from the style embedding, expressed in a style other than that represented by the style embedding of the text input.
2. The computer-implemented method of claim 1 , wherein the encoder neural network includes a multi-head attention neural network unit for performing disentanglement.
3. The computer-implemented method of claim 1 , further comprising updating model parameters using the subset of the dataset as mini-batches by maximizing a log-likelihood of an input text in a variational autoencoder framework.
4. The computer-implemented method of claim 1 , wherein the generator neural network includes a two-layer unidirectional LSTM neural network having an output dimension equal to a vocabulary size of a vocabulary from which the next word prediction is generated.
5. The computer-implemented method of claim 1 , wherein in a subsequent testing sequence, the method further comprises:
fixing one of the disentangled representations while randomly sampling another one of the disentangled representations; and
decoding the fixed one of the disentangled representations with the randomly sampled other one of the disentangled representations.
6. The computer-implemented method of claim 1 , wherein the input text structures are selected from the group consisting of sentences and paragraphs.
7. The computer-implemented method of claim 1 , further comprising disentangling a new text input, using the trained encoder neural network, into a new content embedding and a new style embedding.
8. The computer-implemented method of claim 7 , further comprising generating a new text output using the generator neural network, based on the new content embedding, to perform at least one of style transfer, conditional text generation for question answering or text summarization, or personalized dialogue systems.
9. The computer-implemented method of claim 8 , wherein generating the new text output is further based on a style embedding that is different from the new style embedding.
10. A system for disentangled data generation, comprising:
a hardware processor; and
a memory that stores:
a dataset including a plurality of pairs, each formed from a given one of a plurality of input text structures and a given one of a plurality of style labels for the plurality of input text structures; and
computer program code which, when executed by the hardware processor, implements:
training code that trains an encoder neural network to disentangle a sequential text input into disentangled representations, including a content embedding and a style embedding, based on a subset of the dataset, using an objective function that includes a regularization term that minimizes mutual information between the content embedding and the style embedding, and that trains a generator neural network to generate a text output that includes content from the style embedding, expressed in a style other than that represented by the style embedding of the text input, wherein the objective function is:
VAE +λ reg
wherein λ is a hyperparameter reweighting a regularization reg and a variational autoencoder objective VAE , where reg is expressed as Dis +MI(s; c), including a disentanglement loss Dis and a mutual information term MI(s; c) base on a style embedding s and a content embedding c.
11. The system of claim 10 , wherein the encoder neural network includes a multi-head attention neural network unit for performing disentanglement.
12. The system of claim 10 , wherein the training code further updates model parameters using the subset of the dataset as mini-batches by maximizing a log-likelihood of an input text in a variational autoencoder framework.
13. The system of claim 10 , wherein the generator neural network includes a two-layer unidirectional LSTM neural network having an output dimension equal to a vocabulary size of a vocabulary from which the next word prediction is generated.
14. The system of claim 10 , wherein the training code further:
fixes one of the disentangled representations while randomly sampling another one of the disentangled representations; and
decodes the fixed one of the disentangled representations with the randomly sampled other one of the disentangled representations.
15. The system of claim 10 , wherein the input text structures are selected from the group consisting of sentences and paragraphs.
16. The system of claim 10 , wherein the computer program code further implements disentangling code that disentangles a new text input, using the trained encoder neural network, into a new content embedding and a new style embedding.
17. The system of claim 16 , wherein the computer program code further implements generating code that generates a new text output using the generator neural network, based on the new content embedding, to perform at least one of style transfer, conditional text generation for question answering or text summarization, or personalized dialogue systems.
18. The system of claim 17 , wherein the new text output is further based on a style embedding that is different from the new style embedding.