Language model for abstractive summarization
Methods, systems, and computer programs are presented for abstractive summarization of text by viewing sequence transduction as a language modeling problem. One method comprises an operation for training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for the text being summarized. The method further comprises operations for detecting the text to be summarized, initializing the running summary, and performing a plurality of iterations. Each iteration comprises providing, to the machine-learning model, the source text and the running summary, and adding, using the machine-learning model, a new word to the running summary. Further, the method comprises an operation for storing, on a memory, the running summary as the summary of the text.
1. A computer-implemented method for creating a summary for text, the method comprising:
training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for the text being summarized;
detecting the text to be summarized;
initializing the running summary;
performing a plurality of iterations, each iteration comprising:
providing, to the machine-learning model, the text and the running summary; and
adding, using the machine-learning model, a new word to the running summary; and
storing, on a memory, the running summary as the summary of the text.
2. The method as recited in claim 1 , wherein the training is based on training data, the training data comprising a plurality of conversations and corresponding summaries.
3. The method as recited in claim 2 , wherein the machine-learning program is trained using maximum likelihood, wherein the training data comprises, for each conversation from the plurality of conversations, the conversation, a control token, and the summary of the conversation, the control token separating the conversation from the summary.
4. The method as recited in claim 1 , wherein the machine-learning program is a decoder-only deep-learning transformer.
5. The method as recited in claim 4 , wherein the decoder-only deep-learning transformer comprises four layers comprising: a masked self attention layer, a first norm layer, a feed forward layer, and a second norm layer.
6. The method as recited in claim 1 , wherein the text is embedded using data-driven subword encoding via Byte Pair Encoding (BPE).
7. The method as recited in claim 1 , wherein initializing the running summary comprises setting the running summary to be empty.
8. The method as recited in claim 1 , wherein the text is a conversation that comprises one or more turns.
9. The method as recited in claim 1 , wherein the text is a turn within a conversation.
10. The method as recited in claim 1 , further comprising:
causing presentation of the summary on a display.
11. A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the system to perform operations comprising:
training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for text being summarized;
detecting the text to be summarized;
initializing the running summary;
performing a plurality of iterations, each iteration comprising:
providing, to the machine-learning model, the text and the running summary; and
adding, using the machine-learning model, a new word to the running summary; and
storing, on the memory, the running summary as a summary of the text.
12. The system as recited in claim 11 , wherein the training is based on training data, the training data comprising a plurality of conversations and corresponding summaries.
13. The system as recited in claim 12 , wherein the machine-learning program is trained using maximum likelihood, wherein the training data comprises, for each conversation from the plurality of conversations, the conversation, a control token, and the summary of the conversation, the control token separating the conversation from the summary.
14. The system as recited in claim 11 , wherein the machine-learning program is a decoder-only deep-learning transformer.
15. The system as recited in claim 14 , wherein the decoder-only deep-learning transformer comprises four layers comprising: a masked self attention layer, a first norm layer, a feed forward layer, and a second norm layer.
16. A tangible machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
training a machine-learning program to create a machine-learning model that estimates a word to be added to a running summary for text being summarized;
detecting the text to be summarized;
initializing the running summary;
performing a plurality of iterations, each iteration comprising:
providing, to the machine-learning model, the text and the running summary; and
adding, by the machine-learning model, a new word to the running summary; and
storing, on a memory, the running summary as a summary of the text.
17. The tangible machine-readable storage medium as recited in claim 16 , wherein the training is based on training data, the training data comprising a plurality of conversations and corresponding summaries.
18. The tangible machine-readable storage medium as recited in claim 17 , wherein the machine-learning program is trained using maximum likelihood, wherein the training data comprises, for each conversation from the plurality of conversations, the conversation, a control token, and the summary of the conversation, the control token separating the conversation from the summary.
19. The tangible machine-readable storage medium as recited in claim 16 , wherein the machine-learning program is a decoder-only deep-learning transformer.
20. The tangible machine-readable storage medium as recited in claim 19 , wherein the decoder-only deep-learning transformer comprises four layers comprising: a masked self attention layer, a first norm layer, a feed forward layer, and a second norm layer.