Deep reinforced model for abstractive summarization
A system for text summarization includes an encoder for encoding input tokens of a document and a decoder for emitting summary tokens which summarize the document based on the encoded input tokens. At each iteration the decoder generates attention scores between a current hidden state of the decoder and previous hidden states of the decoder, generates a current decoder context from the attention scores and the previous hidden states of the decoder, and selects a next summary token based on the current decoder context. The selection of the next summary token prevents emission of repeated summary phrases in a summary of the document.
1. A text summarization system comprising:
an encoder for encoding input tokens of a document to be summarized; and
a decoder for emitting summary tokens which summarize the document based on the encoded input tokens, wherein at each iteration the decoder:
generates attention scores between a current hidden state of the decoder and previous hidden states of the decoder;
generates a current decoder context from the attention scores and the previous hidden states of the decoder; and
selects a next summary token based on the current decoder context;
wherein the selection of the next summary token prevents emission of repeated summary phrases in a summary of the document.
2. The text summarization system of claim 1 , wherein the attention scores include an attention score for each of the previous hidden states of the decoder.
3. The text summarization system of claim 1 , wherein at each iteration, the decoder further normalizes the attention scores.
4. The text summarization system of claim 3 , wherein the decoder normalizes the attention scores using a softmax layer.
5. The text summarization system of claim 1 , wherein the current decoder context is a convex combination of the attention scores and the previous hidden states of the decoder.
6. The text summarization system of claim 1 , wherein the next summary token is further based on the current hidden state of the decoder.
7. A method for summarizing text, the method comprising:
receiving a document to be summarized;
encoding, using an encoder, input tokens of the document;
generating, using a decoder, attention scores between a current hidden state of the decoder and previous hidden states of the decoder;
generating, using the decoder, a current decoder context from the attention scores and the previous hidden states of the decoder; and
selecting, using the decoder, a next summary token based on the current decoder context;
wherein:
the next summary token from each iteration summarizes the document; and
the selection of the next summary token prevents emission of repeated summary phrases in a summary of the document.
8. The method of claim 7 , wherein the attention scores include an attention score for each of the previous hidden states of the decoder.
9. The method of claim 7 , further comprising, at each iteration, normalizing the attention scores.
10. The method of claim 9 , wherein normalizing the attention scores comprises using a softmax layer.
11. The method of claim 7 , wherein generating the current decoder context comprises generating a convex combination of the attention scores and the previous hidden states of the decoder.
12. The method of claim 7 , wherein selecting the next summary token is further based on the current hidden state of the decoder.
13. A tangible non-transitory computer readable storage medium impressed with computer program instructions that, when executed on a processor, implement a method comprising:
receiving a document to be summarized;
encoding, using an encoder, input tokens of the document;
generating, using a decoder, attention scores between a current hidden state of the decoder and previous hidden states of the decoder;
generating, using the decoder, a current decoder context from the attention scores and the previous hidden states of the decoder; and
selecting, using the decoder, a next summary token based on the current decoder context;
wherein:
the next summary token from each iteration summarizes the document; and
the selection of the next summary token prevents emission of repeated summary phrases in a summary of the document.
14. The tangible non-transitory computer readable storage medium of claim 13 , wherein the attention scores include an attention score for each of the previous hidden states of the decoder.
15. The tangible non-transitory computer readable storage medium of claim 13 , further comprising, at each iteration, normalizing the attention scores.
16. The tangible non-transitory computer readable storage medium of claim 13 , wherein generating the current decoder context comprises generating a convex combination of the attention scores and the previous hidden states of the decoder.
17. The tangible non-transitory computer readable storage medium of claim 13 , wherein selecting the next summary token is further based on the current hidden state of the decoder.