Systems and methods for long document summarization
Embodiments described herein provide methods and systems for effectively and efficiently summarizing long documents. A transformer is provided with bottom-up and top-down inference combined to effectively capture long-range dependency. In the bottom-up inference, each token only attends to nearby tokens within a window of a specified size. In the top-down inference, full self-attention is given using units with coarser granularity. The bottom-up-inferred token representations are then updated with the top-down representations, which is achieved with cross-attention between the top and token levels. Multiple levels of top-down representations with increasingly coarser granularity can be used if documents are extremely long.
1. A method for document summarization, the method comprising:
receiving, via a communication interface, a text document comprising a plurality of tokens;
computing, at an attention layer, a first set of token representations by attending the plurality of tokens to respective nearby tokens within a pre-defined encoding window;
generating, by a pooling layer, a set of segment representations from the computed first set of token representations from the attention layer;
updating the set of segment representations with a full self-attention layer;
generating a second set of token representations by applying cross attention upon the first set of token representations and the updated set of segment representations after the full self-attention layer; and
sending the generated second set of token representations of the text document to a decoder for generating a summary output based on the second set of token representations.
2. The method of claim 1 , wherein the first set of token representations are generated in a bottom-up manner via a transformer with only local self-attention within a predefined window of each input token.
3. The method of claim 1 , wherein the set of segment representations are top-level representations generated by pooling the first set of token representations, and passing the set of segment representations through a transformer with full self-attention.
4. The method of claim 1 , wherein the cross attention is applied by:
inputting the updated set of segment representations as keys and values to a transformer layer; and
inputting the first set of token representations as queries to the transformer layer.
5. The method of claim 1 wherein the set of segment representations are generated by:
computing a weighted average of tokens within respective subsets of the first set of token representations.
6. The method of claim 1 , further comprising:
generating, via a pooling layer, a set of intermediate representations from the first set of token representations; and
applying local self-attention to the set of intermediate representations,
wherein the set of intermediate representations are pooled to form at least a part of the set of segment representations.
7. The method of claim 1 , further comprising:
updating the first set of token representations with local self-attention to produce a third set of token representations,
wherein the updating the first set of token representations with cross-attention comprises updating the third set of token representations with cross-attention between the third set of token representations and the set of segment representations.
8. A system for document summarization, the system comprising:
a memory that stores a summarization model;
a communication interface that receives a text document comprising a plurality of tokens; and
one or more hardware processors that:
computes, at an attention layer, a first set of token representations by attending the plurality of tokens to respective nearby tokens within a pre-defined encoding window;
generates, by a pooling layer, a set of segment representations from the computed first set of token representations from the attention layer;
updates the set of segment representations with a full self-attention layer;
generates a second set of token representations by applying cross attention upon the first set of token representations and the updated set of segment representations after the full self-attention layer; and
sends the generated second set of token representations of the text document to a decoder for generating a summary output based on the second set of token representations.
9. The system of claim 8 , wherein the first set of token representations are generated in a bottom-up manner via a transformer with only local self-attention within a predefined window of each input token.
10. The system of claim 8 , wherein the set of segment representations are top-level representations generated by pooling the first set of token representations, and passing the set of segment representations through a transformer with full self-attention.
11. The system of claim 8 , wherein the cross attention is applied by:
inputting the updated set of segment representations as keys and values to a transformer layer; and
inputting the first set of token representations as queries to the transformer layer.
12. The system of claim 8 wherein the set of segment representations are generated by:
computing a weighted average of tokens within respective subsets of the first set of token representations.
13. The system of claim 8 , wherein the one or more hardware processors further:
generates, via a pooling layer, a set of intermediate representations from the first set of token representations; and
applies local self-attention to the set of intermediate representations,
wherein the set of intermediate representations are pooled to form at least a part of the set of segment representations.
14. The system of claim 8 , wherein the one or more hardware processors further:
updates the first set of token representations with local self-attention to produce a third set of token representations,
wherein the updating the first set of token representations with cross-attention comprises updating the third set of token representations with cross-attention between the third set of token representations and the set of segment representations.
15. A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions for training a multi-document summarization model, the instructions being executed by at least one processor to perform operations comprising:
receiving, via a communication interface, a text document comprising a plurality of tokens;
computing, at an attention layer, a first set of token representations by attending the plurality of tokens to respective nearby tokens within a pre-defined encoding window;
generating, by a pooling layer, a set of segment representations from the computed first set of token representations from the attention layer;
updating the set of segment representations with a full self-attention layer;
generating a second set of token representations by applying cross attention upon the first set of token representations and the updated set of segment representations after the full self-attention layer; and
sending the generated second set of segment representations of the text document to a decoder for generating a summary output based on the second set of segment representations.
16. The processor-readable non-transitory storage medium of claim 15 , wherein the first set of token representations are generated in a bottom-up manner via a transformer with only local self-attention within a predefined window of each input token.
17. The processor-readable non-transitory storage medium of claim 15 , wherein the set of segment representations are top-level representations generated by pooling the first set of token representations, and passing the set of segment representations through a transformer with full self-attention.
18. The processor-readable non-transitory storage medium of claim 15 , wherein the cross attention is applied by:
inputting the updated set of segment representations as keys and values to a transformer layer; and
inputting the first set of token representations as queries to the transformer layer.
19. The processor-readable non-transitory storage medium of claim 15 wherein the set of segment representations are generated by:
computing a weighted average of tokens within respective subsets of the first set of token representations.
20. The processor-readable non-transitory storage medium of claim 15 , the instructions being executed by a processor to perform operations further comprising:
generating, via a pooling layer, a set of intermediate representations from the first set of token representations; and
applying local self-attention to the set of intermediate representations,
wherein the set of intermediate representations are pooled to form at least a part of the set of segment representations.