Systems and methods for text summarization
Embodiments described herein provide a method for text summarization. The method includes receiving a training dataset having at least an uncompressed text, a compressed text, and one or more information entities accompanying the compressed text. The method also includes generating, using a perturber model, a perturbed text with the one or more information entities being inserted into the compressed text. The method further includes training the perturber model based on a first training objective, and generating, using the trained perturber model, a perturbed summary in response to an input of a reference summary. The method further includes generating, via an editor model, a predicted summary by removing information from the perturbed summary conditioned on a source document of the reference summary, and training the editor model based on a second training objective.
1. A method for text summarization, the method comprising:
receiving, via a communication interface, a training dataset comprising at least an uncompressed text, a compressed text, and one or more information entities that are not mentioned in the compressed text but included in the uncompressed text;
generating, by a perturber model, a perturbed text based on inserting at least a first information entity from the one or more information entities into the compressed text;
training the perturber model based on a first training objective comparing the perturbed text generated from the compressed text and the at least first information entity, and the uncompressed text including the one or more information entities;
generating, by the trained perturber model, a perturbed summary in response to an input of a reference summary and at least a second information entity randomly selected from a corresponding source document of the reference summary;
generating, via an editor model, a predicted summary by removing information from the perturbed summary conditioned on the source document of the reference summary; and
training the editor model based on a second training objective of a cross-entropy loss between a predicted token distribution of the predicted summary conditioned on the source document and a token distribution of the reference summary.
2. The method of claim 1 , wherein the one or more information entities are not mentioned in the compressed text.
3. The method of claim 1 , wherein the information removed from the predicted summary comprises a plurality of information entities not mentioned in the source document.
4. The method of claim 1 , wherein the first training objective is a cross-entropy loss between a predicted token distribution of the perturbed text and a token distribution of the uncompressed text.
5. The method of claim 1 , wherein the perturbed summary is generated by inserting information into the reference summary, and wherein at least a portion of the inserted information entities is randomly selected from the source document but not present in the reference summary.
6. The method of claim 1 , wherein the perturbed summary has a format of a sequence of tokens with hashtags identifying inserted tokens corresponding to inserted one or more information entities that do not belong to the reference summary.
7. The method of claim 1 , wherein the training the editor model further comprises updating the editor model based on the cross-entropy loss via backpropagation.
8. The method of claim 1 , further comprising:
receiving, at the trained editor model, a testing summary and a test source document; and
generating, by the trained editor model, a pruned summary conditioned on the test source document.
9. The method of claim 1 , wherein:
the perturber model comprises a denoising autoencoder; and
the editor model comprises a denoising autoencoder.
10. The method of claim 1 , wherein the one or more information entities are randomly selected from a pool of information entities that is overlapping with the uncompressed text and non-overlapping with the compressed text.
11. The method of claim 1 , wherein the generating of the perturbed text with the one or more information entities comprises the inserting of one or more information entities into the compressed text without swapping with the one or more information entities.
12. The method of claim 1 , wherein the generating of the predicted summary comprises the removing of the information entities from the perturbed summary without swapping the information.
13. The method of claim 1 , wherein:
training data of the perturber model comprises a set of sentence-compression data, and the set of sentence-compression data comprises one or more pairs of a compressed text, a corresponding uncompressed, and one or more perturber information entities; and
the uncompressed text contains the compressed text and the one or more information entities.
14. The method of claim 1 , wherein a number of tokens in the uncompressed text is more a number of tokens in the compressed text by up to one third of the number of tokens in the compressed text.
15. A system for text summarization, the system comprising:
a communication interface that receives a training dataset comprising at least an uncompressed text, a compressed text, and one or more information entities that are not mentioned in the compressed text;
a memory storing a plurality of processor-executable instructions; and
a processor executing the instructions to perform operations comprising:
generating, by a perturber model, a perturbed text based on inserting at least a first information entity from the one or more information entities into the compressed text;
training the perturber model based on a first training objective comparing the perturbed text generated from the compressed text and the at least first information entity and the uncompressed text including the one or more information entities;
generating, by the trained perturber model, a perturbed summary in response to an input of a reference summary and at least a second information entity randomly selected from a corresponding source document of the reference summary;
generating, via an editor model, a predicted summary by removing information from the perturbed summary conditioned on the source document of the reference summary; and
training the editor model based on a second training objective of a cross-entropy loss between a predicted token distribution of the predicted summary conditioned on the source document and a token distribution of the reference summary.
16. The system of claim 15 , wherein the one or more information entities are not mentioned in the compressed text.
17. The system of claim 15 , wherein the information removed from the predicted summary comprises a plurality of information entities not mentioned in the source document.
18. The system of claim 15 , wherein the first training objective is a cross-entropy loss between a predicted token distribution of the perturbed text and a token distribution of the uncompressed text.
19. The system of claim 15 , wherein the perturbed summary is generated by inserting information into the reference summary, and wherein at least a portion of the inserted information entities is randomly selected from the source document but not present in the reference summary.
20. The system of claim 15 , wherein the perturbed summary has a format of a sequence of tokens with hashtags identifying inserted tokens corresponding to inserted one or more information entities that do not belong to the reference summary.
21. The system of claim 15 , wherein the training the editor model further comprises updating the editor model based on the cross-entropy loss via backpropagation.
22. The system of claim 15 , further comprising:
receiving, at the trained editor model, a testing summary and a test source document; and
generating, by the trained editor model, a pruned summary conditioned on the test source document.
23. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for code program synthesis for a target problem, the instructions being executed by one or more hardware processors to perform operations comprising:
receiving, via a communication interface, a training dataset comprising at least an uncompressed text, a compressed text, and one or more information entities that are not mentioned in the compressed text;
generating, by a perturber model, a perturbed text based on inserting at least a first information entity from the one or more information entities into the compressed text;
training the perturber model based on a first training objective comparing the perturbed text generated from the compressed text and the at least first information entity and the uncompressed text including the one or more information entities;
generating, by the trained perturber model, a perturbed summary in response to an input of a reference summary and at least a second information entity randomly selected from a corresponding source document of the reference summary;
generating, via an editor model, a predicted summary by removing information from the perturbed summary conditioned on the source document of the reference summary; and
training the editor model based on a second training objective of a cross-entropy loss between a predicted token distribution of the predicted summary conditioned on the source document and a token distribution of the reference summary.