Abstractive sentence summarization
In one embodiment, a sequence of input words is received. Each of the input words is encoded as an indicator vector, wherein a sequence of the indicator vectors captures features of the sequence of input words. The sequence of the indicator vectors is then mapped to a distribution of a contextual probability of a first output word in a sequence of output words. For each subsequent output word, the sequence of the indicator vectors is encoded with a context, wherein the context comprises a previously mapped contextual probability distribution of a fixed window of previous output words; and the encoded sequence of the indicator vectors and the context is mapped to the distribution of the contextual probability of the subsequent output word. Finally, a condensed summary is generated using a decoder by maximizing the contextual probability of each of the output words.
1. A method comprising, by one or more computer server devices:
receiving an input sentence comprising a sequence of input words, wherein a set of words in a vocabulary comprises the input words;
encoding each of the input words as an indicator vector, wherein the indicator vector captures features of the input word and a sequence of the indicator vectors captures features of the sequence of the input words;
mapping, using a neural network language model, the sequence of the indicator vectors to a distribution of a contextual probability of a first output word in a sequence of output words, wherein the set of words in the vocabulary comprises the output words;
for each of one or more next output words of the sequence of output words:
encoding the sequence of the indicator vectors with a context, wherein the context comprises a previously mapped contextual probability distribution of a fixed window of previous ones of the output words; and
mapping the encoded sequence of the indicator vectors and the context to the distribution of the contextual probability of the next output word; and
generating a condensed summary using a decoder by maximizing the contextual probability of each of the output words in the sequence of the output words.
2. The method of claim 1 , wherein the encoding comprises using an attention-based encoder which is used to find a latent soft alignment between the indicator vectors and the context, and wherein the latent soft alignment points to a position in the sequence of the indicator vectors where a block of highly relevant information for generating the summary is concentrated.
3. The method of claim 1 , wherein a number of the output words in the sequence of the output words is pre-determined.
4. The method of claim 1 , wherein the decoder is a Viterbi decoder that finds an exact solution by searching through an entire distribution of the contextual probability.
5. The method of claim 1 , wherein the decoder is a beam search decoder that finds an approximate solution by searching through a limited distribution of the contextual probability.
6. The method of claim 1 , further comprising modifying a scoring function to find extractive word matches from the input sentences by directly estimating the contextual probability using a log-linear model.
7. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive an input sentence comprising a sequence of input words, wherein a set of words in a vocabulary comprises the input words;
encode each of the input words as an indicator vector, wherein the indicator vector captures features of the input word and a sequence of the indicator vectors captures features of the sequence of the input words;
map, using a neural network language model, the sequence of the indicator vectors to a distribution of a contextual probability of a first output word in a sequence of output words, wherein the set of words in the vocabulary comprises the output words;
for each of one or more next output words of the sequence of output words:
encode the sequence of the indicator vectors with a context, wherein the context comprises a previously mapped contextual probability distribution of a fixed window of previous ones of the output words; and
map the encoded sequence of the indicator vectors and the context to the distribution of the contextual probability of the next output word; and
generate a condensed summary using a decoder by maximizing the contextual probability of each of the output words in the sequence of the output words.
8. The media of claim 7 , wherein the encoding comprises using an attention-based encoder which is used to find a latent soft alignment between the indicator vectors and the context, and wherein the latent soft alignment points to a position in the sequence of the indicator vectors where a block of highly relevant information for generating the summary is concentrated.
9. The media of claim 7 , wherein a number of the output words in the sequence of the output words is pre-determined.
10. The media of claim 7 , wherein the decoder is a Viterbi decoder that finds an exact solution by searching through an entire distribution of the contextual probability.
11. The media of claim 7 , wherein the decoder is a beam search decoder that finds an approximate solution by searching through a limited distribution of the contextual probability.
12. The media of claim 7 , further embodying software that is operable when executed to:
modify a scoring function to find extractive word matches from the input sentences by directly estimating the contextual probability using a log-linear model.
13. A system comprising: one or more processors; and a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to:
receive an input sentence comprising a sequence of input words, wherein a set of words in a vocabulary comprises the input words;
encode each of the input words as an indicator vector, wherein the indicator vector captures features of the input word and a sequence of the indicator vectors captures features of the sequence of the input words;
map, using a neural network language model, the sequence of the indicator vectors to a distribution of a contextual probability of a first output word in a sequence of output words, wherein the set of words in the vocabulary comprises the output words;
for each of one or more next output words of the sequence of output words:
encode the sequence of the indicator vectors with a context, wherein the context comprises a previously mapped contextual probability distribution of a fixed window of previous ones of the output words; and
map the encoded sequence of the indicator vectors and the context to the distribution of the contextual probability of the next output word; and
generate a condensed summary using a decoder by maximizing the contextual probability of each of the output words in the sequence of the output words.
14. The system of claim 13 , wherein the encoding comprises using an attention-based encoder which is used to find a latent soft alignment between the indicator vectors and the context, and wherein the latent soft alignment points to a position in the sequence of the indicator vectors where a block of highly relevant information for generating the summary is concentrated.
15. The system of claim 13 , wherein a number of the output words in the sequence of the output words is pre-determined.
16. The system of claim 13 , wherein the decoder is a Viterbi decoder that finds an exact solution by searching through an entire distribution of the contextual probability.
17. The system of claim 13 , wherein the decoder is a beam search decoder that finds an approximate solution by searching through a limited distribution of the contextual probability.
18. The system of claim 13 , wherein the processors are further operable when executing the instructions to:
modify a scoring function to find extractive word matches from the input sentences by directly estimating the contextual probability using a log-linear model.