Systems and methods for generating summaries of documents
Systems and methods for summarizing online articles for consumption on a user device are disclosed herein. The system extracts the main body of an article's text from the HTML code of an online article. The system may then classify the extracted article into one of several different categories and removes duplicate articles. The system breaks down the article into its component sentences, and each sentence is classified into one of three categories: (1) potential candidate sentences that may be included in the generated summary; (2) weakly rejected sentences that will not be included in the summary but may be used to generate the summary; and (3) strongly rejected sentences that are not included in the summary. Finally, the system applies a document summarizer to generate quickly readable article summaries, for viewing on the user device, using relevant sentences from the article while maintaining the coherence of the article.
1. A system, comprising:
an interface processor that provides an interactive graphical user interface to a user device over a network;
a parser processor that retrieves an RSS feed over the network, and generate an initial set of articles based on the RSS feed;
a categorization processor that categorizes each article in the initial set of articles into one of a plurality of subject matter categories;
a deduplication processor that generates a final set of articles by removing duplicate articles from the initial set of articles, wherein each article in the final set of articles comprises a plurality of sentences and a title; and
a summarization processor that, for each article in the final set of articles:
generates a preliminary score for each sentence in the plurality of sentences,
assigns each sentence in the plurality of sentences to one of three categories,
generates an article summary, wherein the article summary comprises one or more sentences from one of the three categories, wherein the article summary is based at least in part on the preliminary score for each sentence, and
provides the article summary to the user device over the network via the interface processor;
wherein the three categories consist of strongly rejected sentences, weakly rejected sentences, and potential candidate sentences.
2. The system of claim 1 , wherein the summarization processor assigns a sentence in the plurality of sentences to the weakly rejected sentences category based at least in part on one or more weak linguistic markers in the sentence, wherein the one or more weak linguistic markers include personal pronouns, demonstrative pronouns, adversative conjunctions, concessive conjunctions, coordinating conjunctions, adverbial discourse connectors, deictic metadiscourse connectors, temporal adverbs, and no verbs.
3. The system of claim 1 , wherein the summarization processor assigns a sentence in the plurality of sentences to the strongly rejected sentences category based at least in part on one or more strong linguistic markers in the sentence, wherein the one or more strong linguistic markers include gratitude nouns, gratitude verbs, greetings, exclamations, and hypertext jargon.
4. The system of claim 1 , wherein the summarization processor generates an article summary by:
1) determining a maximal marginal relevance score for each sentence in the potential candidate sentences category based at least in part on the preliminary score for that sentence, the title of the article, a vector space of the combination of the sentences in the potential candidate sentences category and the sentences in the weakly rejected sentences category, and a weighting constant;
2) adding the sentence with the highest maximal marginal relevance score to the article summary;
3) removing the sentence with the highest maximal marginal relevance score from the potential candidate sentences category; and
4) repeating steps 1-3 until the number of characters in the article summary reaches a predetermined threshold.
5. The system of claim 4 , wherein the summarization processor reorders the sentences in the article summary based on the sentence's position in the article.
6. The system of claim 4 , wherein each article summary is categorized into the same subject matter category as the corresponding article.
7. A method, comprising:
retrieving, using a network-enabled computer, an RSS feed, wherein the RSS feed comprises a collection of articles;
generating an initial set of articles based on the RSS feed;
categorizing each article in the initial set of articles into one of a plurality of subject matter categories;
generating a final set of articles by removing duplicate articles from the initial set of articles, wherein each article in the final set of articles comprises a plurality of sentences; and
for each article in the final set of articles:
generating a preliminary score for each sentence in the plurality of sentences,
assigning each sentence in the plurality of sentences to one of three categories,
generating an article summary, wherein the article summary comprises one or more sentences from one of the three categories, wherein the article summary is based at least in part on the preliminary score for each sentence, and
providing the article summary to a user device over a network;
wherein the three categories consist of strongly rejected sentences, weakly rejected sentences, and potential candidate sentences.
8. The method of claim 7 , assigning each sentence in the plurality of sentences to one of three categories comprises assigning a sentence in the plurality of sentences to the weakly rejected sentences category based at least in part on one or more weak linguistic markers in the sentence, wherein the one or more weak linguistic markers include personal pronouns, demonstrative pronouns, adversative conjunctions, concessive conjunctions, coordinating conjunctions, adverbial discourse connectors, deictic metadiscourse connectors, temporal adverbs, and no verbs.
9. The method of claim 7 , assigning each sentence in the plurality of sentences to one of three categories comprises assigning a sentence in the plurality of sentences to the strongly rejected sentences category based at least in part on one or more strong linguistic markers in the sentence, wherein the one or more strong linguistic markers include gratitude nouns, gratitude verbs, greetings, exclamations, and hypertext jargon.
10. The method of claim 7 , wherein generating an article summary comprises:
1) determining a maximal marginal relevance score for each sentence in the potential candidate sentences category based at least in part on the preliminary score for that sentence, the title of the article, a vector space of the combination of the sentences in the potential candidate sentences category and the sentences in the weakly rejected sentences category, and a weighting constant;
2) adding the sentence with the highest maximal marginal relevance score to the article summary;
3) removing the sentence with the highest maximal marginal relevance score from the potential candidate sentences category; and
4) repeating steps 1-3 until the number of characters in the article summary reaches a predetermined threshold.
11. The method of claim 10 , further comprising reordering the sentences in the article summary based on the sentence's position in the article.
12. The method of claim 10 , further comprising categorizing each article summary into the same subject matter category as the corresponding article.