Title extraction using natural language processing
Methods, systems, and computer programs are presented for selecting a title for a post of a social network. A method includes operations for identifying a word weight for each word in a corpus based on the relevance of each word, receiving a document that includes several sentences, and assigning to each word in the sentences a tag indicative of a part of speech. Further, the method includes operations for calculating a topic weight for each sentence based on the tags of the words in the sentence, and for calculating a sentence weight for each sentence based on a sum of the word weights of the words in the sentence, the topic weight for the sentence, and a position weight of the sentence within a paragraph. The sentence with the highest sentence weight is selected as the title for the document, and the title is presented with the document to a user.
1. A method comprising:
identifying a word weight for each word in a corpus based on a relevance of each word;
receiving a document that comprises a plurality of sentences;
assigning to each word in the plurality of sentences a tag indicative of a part of speech of the word;
calculating a topic weight for each sentence based on the tags of the words in the sentence, wherein calculating the topic weight further comprises:
calculating an event weight for the sentence based on the tags of the words in the sentence;
calculating an entity weight for the sentence based on the tags of the words in the sentence; and
calculating a sentiment weight for the sentence based on the tags of the words in the sentence;
calculating a sentence weight for each sentence based on a sum of the word weights of the words in the sentence, the topic weight for the sentence, and a position weight of the sentence within a paragraph;
selecting the sentence with a highest sentence weight as a title for the document; and
causing the title to be presented with the document to a user.
2. The method as recited in claim 1 , wherein calculating the topic weight further comprises:
calculating the topic weight of the sentence as a weighted sum of the event weight, the entity weight, and the sentiment weight of the sentence.
3. The method as recited in claim 1 , wherein calculating the event weight further comprises:
extracting events from the sentence, wherein each event includes a verb or a verb and a noun; and
calculating the event weight of the sentence as a sum of the word weights of the words in the extracted events.
4. The method as recited in claim 1 , wherein calculating the entity weight further comprises:
extracting entities from the sentence, wherein each entity includes a noun or an adjective and a noun; and
calculating the entity weight of the sentence as a sum of the word weights of the words in the extracted entities.
5. The method as recited in claim 1 , wherein calculating the sentiment weight further comprises:
extracting sentiments from the sentence, wherein each sentiment includes an adjective; and
calculating the sentiment weight of the sentence as a sum of the word weights of the words in the extracted sentiments.
6. The method as recited in claim I. wherein the assigning to each word the tag further comprises:
utilizing a natural language processor to determine a part of speech to the word; and
assigning the tag based on the determined part of speech.
7. The method as recited in claim I, wherein the identifying the word weight further comprises:
identifying words in the corpus;
stemming the words in the corpus;
determining a frequency of occurrence in the corpus for each stemmed word; and
assigning the word weight to the stemmed words based on the frequency of occurrence.
8. The method as recited in claim 1 , wherein calculating the sentence weight for each sentence further comprises:
calculating the position weight for the sentence as a maximum between a first constant and a second constant minus a position of the sentence within the paragraph times a third constant.
9. The method as recited in claim 8 , wherein calculating the sentence weight for each sentence further comprises:
calculating the sentence weight as the position weight times a weighted sum of the topic weight and the sum of the word weights of the words in the sentence.
10. A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
identifying a word weight for each word in a corpus based on a relevance of each word;
receiving a document that comprises a plurality of sentences;
assigning to each word in the plurality of sentences a tag indicative of a part of speech of the word;
calculating a topic weight for each sentence based on the tags of the words in the sentence, wherein calculating the topic weight further comprises:
calculating an event weight for the sentence based on the tags of the words in the sentence;
calculating an entity weight for the sentence based on the tags of the words in the sentence; and
calculating a sentiment weight for the sentence based on the tags of the words in the sentence;
calculating a sentence weight for each sentence based on a sum of the word weights of the words in the sentence, the topic weight for the sentence, and a position weight of the sentence within a paragraph;
selecting the sentence with a highest sentence weight as a title for the document; and
causing the title to be presented with the document to a user.
11. The system as recited in claim 10 , wherein calculating the topic weight further comprises:
calculating the topic weight of the sentence as a weighted sum of the event weight, the entity weight, and the sentiment weight of the sentence.
12. A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
identifying a word weight for each word in a corpus based on a relevance of each word;
receiving a document that comprises a plurality of sentences;
assigning to each word in the plurality of sentences a tag indicative of a part of speech of the word;
calculating a topic weight for each sentence based on the tags of the words in the sentence;
calculating a sentence weight for each sentence based on a sum of the word weights of the words in the sentence the topic weight for the sentence, and a position weight of the sentence within a paragraph, wherein calculating the sentence weight for each sentence further comprises calculating the position weight for the sentence as a maximum between a first constant and a second constant minus a position of the sentence within the paragraph times a third constant;
selecting, the sentence with a highest sentence weight as a title for the document; and
causing the title to be presented with the document to a user.
13. The system as recited in claim 12 , wherein calculating the sentence weight for each sentence further comprises:
calculating the sentence weight as the position weight times a weighted sum of the topic weight and the sum of the word weights of the words in the sentence.
14. A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
identifying a word weight for each word in a corpus based on a relevance of each word;
receiving a document that comprises a plurality of sentences;
assigning to each word in the plurality of sentences a tag indicative of a part of speech of the word;
calculating a topic weight for each sentence based on the tags of the words in the sentence, wherein calculating the topic weight further comprises:
calculating an event weight for the sentence based on the tags of the words in the sentence;
calculating an entity weight for the sentence based on the tags of the words in the sentence;
calculating a sentiment weight for the sentence as on the tags of the words in the sentence; and
calculating the topic weight of the sentence as a weighted SUM of the event weight, the entity weight, and the sentiment weight of the sentence;
calculating a sentence weight for each sentence based on a sum of the word weights of the words in the sentence, the topic weight for the sentence, and a position weight of the sentence within a paragraph;
selecting the sentence with a highest sentence weight as a title for the document; and
causing the title to be presented with the document to a user.
15. The machine-readable storage medium as recited in claim 14 , wherein calculating the event weight further comprises:
extracting events from the sentence, wherein each event includes a verb or a verb and a noun; and
calculating the event weight of the sentence as a sum of the word weights of the words in the extracted events.
16. The machine-readable storage medium as recited in claim 14 , wherein calculating the entity weight further comprises:
extracting entities from the sentence, wherein each entity includes a noun or an adjective and a noun; and
calculating the entity weight of the sentence as a sum of the word weights of the words in the extracted entities.
17. The machine-readable storage medium as recited in claim 14 , wherein calculating the sentiment weight further comprises:
extracting sentiments from the sentence, wherein each sentiment includes an adjective; and
calculating the sentiment weight of the sentence as a sum of the word weights of the words in the extracted sentiments.