Systems and methods for abstractive document summarization with entity coverage control
Embodiments described herein provide document summarization systems and methods that utilize fine-tuning of pre-trained abstractive summarization models to produce summaries that more faithfully track the content of the documents. Such abstractive summarization models may be pre-trained using a corpus consisting of pairs of articles and associated summaries. For each article-summary pair, a pseudo label or control code is generated and represents a faithfulness of the summary with respect to the article. The pre-trained model is then fine-tuned based on the article-summary pairs and the corresponding control codes. The resulting fine-tuned models then provide improved faithfulness in document summarization tasks.
1. A method for abstractive summarization of a document, the method comprising:
receiving, via a data interface, a training dataset comprising a plurality of articles and a plurality of summaries corresponding to the plurality of articles;
generating a plurality of article-summary pairs by pairing each article with at least one associated summary;
computing, for an article-summary pair, an entity coverage precision metric based on a number of entity mentions in a corresponding summary or a corresponding article;
determining a pseudo label indicating a faithfulness level of the corresponding article and the corresponding summary based on the computed entity coverage precision metric;
prepending the article with the determined pseudo label as a training input to a summarization model;
generating, by the summarization model, an output summary conditioned on both the article and the prepended pseudo label; and
updating the summarization model based on a training objective comparing the output summary and the corresponding summary.
2. The method of claim 1 , wherein the summarization model is based at least in part on a Bidirectional and Auto-Regressive Transformer (BART) summarization model.
3. The method of claim 1 , wherein the plurality of articles comprise a first article and a second article, each associated with a domain-specific database.
4. The method of claim 1 , wherein the plurality of articles comprises an encyclopedia article.
5. The method of claim 4 , further comprising:
generating the output summary conditioned on the encyclopedia article and the prepended pseudo label.
6. The method of claim 1 , wherein the entity coverage precision metric between the summary and the article is computed as a first number of entities mentioned by both the summary and the article divided by a second number of entities mentioned by the summary.
7. The method of claim 1 , wherein the determining the pseudo label indicating the faithfulness level of the corresponding article and the corresponding summary based on the at least one computed entity coverage precision metric for the training dataset comprises:
computing a plurality of entity coverage precision metrics corresponding to the plurality of article-summary pairs;
binning the plurality of entity coverage precision metrics to determine a plurality of binned pseudo labels;
computing an entity coverage precision metric for the corresponding article and the corresponding summary; and
assigning a binned pseudo label of the plurality of binned pseudo labels to the corresponding article and the corresponding summary based on the entity coverage precision metric for the corresponding article and the corresponding summary.
8. A system for abstractive summarization of a document, the system comprising:
a non-transitory memory; and
one or more processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising:
receiving, via a data interface, a training dataset comprising a plurality of articles and a plurality of summaries corresponding to the plurality of articles;
generating a plurality of article-summary pairs by pairing each article with at least one associated summary;
computing, for an article-summary pair, an entity coverage precision metric based on a number of entity mentions in a corresponding summary or a corresponding article;
determining a pseudo label indicating a faithfulness level of the corresponding article and the corresponding summary based on the computed entity coverage precision metric;
prepending the article with the determined pseudo label as a training input to a summarization model;
generating, by the summarization model, an output summary conditioned on both the article and the prepended pseudo label; and
updating the summarization model based on a training objective comparing the output summary and the corresponding summary.
9. The system of claim 8 , the plurality of articles comprise a first article and a second article, each associated with a domain-specific database.
10. The system of claim 8 , wherein the training dataset comprises a plurality of articles and a plurality of summaries each associated with a domain-specific database.
11. The system of claim 8 , wherein the training dataset comprises a plurality of articles and a plurality of summaries each associated with an encyclopedia article.
12. The system of claim 11 , wherein the operations further comprise generating the output summary conditioned on the encyclopedia article and the prepended pseudo label.
13. The system of claim 8 , wherein the entity coverage precision metric between the summary and the article is computed as a first number of entities mentioned by both the summary and the article divided by a second number of entities mentioned by the summary.
14. The system of claim 8 , wherein the determining the pseudo label indicating the faithfulness level of the corresponding article and the corresponding summary based on the at least one computed entity coverage precision metric for the training dataset comprises:
computing a plurality of entity coverage precision metrics corresponding to the plurality of article-summary pairs;
binning the plurality of entity coverage precision metrics to determine a plurality of binned pseudo labels;
computing an entity coverage precision metric for the corresponding article and the corresponding summary; and
assigning a binned pseudo label of the plurality of binned pseudo labels to the corresponding article and the corresponding summary based on the entity coverage precision metric for the corresponding article and the corresponding summary.
15. A non-transitory, machine-readable medium having stored thereon machine-readable instructions executable to cause a system to perform operations comprising:
receiving, via a data interface, a training dataset comprising a plurality of articles and a plurality of summaries associated with the plurality of articles;
generating a plurality of article-summary pairs by pairing each article with at least one associated summary;
computing, for an article-summary pair, an entity coverage precision metric based on a number of entity mentions in a corresponding summary or a corresponding article;
determining a pseudo label indicating a faithfulness level of the corresponding article and the corresponding summary based on the computed entity coverage precision metric;
prepending the article with the determined pseudo label as a training input to a summarization model;
generating, by the summarization model, an output summary conditioned on both the article and the prepended pseudo label; and
updating the summarization model based on a training objective comparing the output summary and the corresponding summary.
16. The non-transitory, machine-readable medium of claim 15 , wherein the training dataset comprises a plurality of articles and a plurality of summaries each associated with a domain-specific database.
17. The non-transitory, machine-readable medium of claim 15 , wherein the training dataset comprises a plurality of articles and a plurality of summaries each associated with an encyclopedia article.
18. The non-transitory, machine-readable medium of claim 17 , wherein the operations further comprise generating the plurality of summaries associated with the encyclopedia article.
19. The non-transitory, machine-readable medium of claim 15 , wherein the entity coverage precision metric between the summary and the article is computed as a first number of entities mentioned by both the summary and the article divided by a second number of entities mentioned by the summary.
20. The non-transitory, machine-readable medium of claim 15 , wherein the determining the pseudo label indicating the faithfulness level of the corresponding article and the corresponding summary based on the at least one computed entity coverage precision metric for the training dataset comprises:
computing a plurality of entity coverage precision metrics corresponding to the plurality of article-summary pairs;
binning the plurality of entity coverage precision metrics to determine a plurality of binned pseudo labels;
computing an entity coverage precision metric for the corresponding article and the corresponding summary; and
assigning a binned pseudo label of the plurality of binned pseudo labels to the corresponding article and the corresponding summary based on the entity coverage precision metric for the corresponding article and the corresponding summary.