Topic extraction using clause segmentation and high-frequency words

1. A method, comprising:

obtaining a set of clauses in a first set of content items comprising unstructured data;

obtaining a set of stop words comprising high-frequency words that occur in a second set of content items;

automatically extracting, by one or more computer systems, a set of topics from the set of clauses by:

generating a set of n-grams from the set of clauses;

excluding a first n-gram from the set of n-grams when the first n-gram contains a word in the set of stop words in a pre-specified position of the first n-gram; and

applying a morphological filter to the set of n-grams to yield a subset of the set of n-grams, wherein:

the set of topics comprises the subset of the set of n-grams; and

the morphological filter is independent of words in the set of stop words; and

displaying, by the one or more computer systems, the set of topics to a user to improve understanding of the first set of content items by the user without requiring the user to manually analyze the first set of content items;

wherein extracting the set of topics from the set of clauses further comprises:

excluding a second n-gram from the set of n-grams when the second n-gram contains a proportion of words in the set of stop words that exceeds a threshold.

2. The method of claim 1 , wherein obtaining the set of stop words comprising high-frequency words that occur in the second set of content items comprises:

ordering a set of words in the second set of content items by frequency; and

including a pre-specified number of most frequent words from the ordered set of words in the set of stop words.

3. The method of claim 1 , further comprising:

generating the set of clauses from the first set of content items by separating a content item into two or more clauses based on a presence of a connective word between the two or more clauses.

4. The method of claim 1 , further comprising:

generating the set of clauses from the first set of content items by separating a content item into two or more clauses based on a presence of a connective punctuation mark between the two or more clauses.

5. The method of claim 1 , further comprising:

identifying a set of synonymous topics within the set of topics; and

merging the synonymous topics under a representative topic.

6. The method of claim 5 , wherein identifying the set of synonymous topics within the set of topics comprises:

obtaining a first set of attributes associated with a first topic in the set of topics and a second set of attributes associated with a second topic in the set of topics;

determining a similarity between the first and second set of attributes; and

identifying the first and second topics as synonymous when the similarity exceeds a threshold.

7. The method of claim 1 , wherein the first set of content items is associated with use of one or more products in an online professional network.

8. The method of claim 1 , wherein the pre-specified position of the n-gram is at least one of:

a beginning of the n-gram; and

an end of the n-gram.

9. The method of claim 1 , further comprising, for each of one or more topics in the set of topics:

automatically extract, from the set of clauses, a set of attributes that provides a context for the topic, wherein the set of attributes comprises a set of words in a vicinity of the topic.

10. The method of claim 9 , further comprising, for each of the one or more topics:

obtain a refined set of attributes by removing from the set of attributes one or more high-frequency words that appear in the set of n-grams; and

use the refined set of attributes to obtain a set of sentiments associated with the topic.

11. An apparatus, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the apparatus to:

obtain a set of clauses in a first set of content items comprising unstructured data;

obtain a set of stop words comprising high-frequency words that occur in a second set of content items;

automatically extract a set of topics from the set of clauses by:

generating a set of n-grams from the set of clauses;

excluding a first n-gram from the set of n-grams when the first n-gram contains a word in the set of stop words in a pre-specified position of the first n-gram; and

applying a morphological filter to the set of n-grams to yield a subset of the set of n-grams, wherein:

the set of topics comprises the subset of the set of n-grams; and

the morphological filter is independent of words in the set of stop words; and

display the set of topics to a user to improve understanding of the first set of content items by the user without requiring the user to manually analyze the first set of content items;

wherein extracting the set of topics from the set of clauses further comprises:

excluding a second n-gram from the set of n-grams when the second n-gram contains a proportion of words in the set of stop words that exceeds a threshold.

12. The apparatus of claim 11 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

generate the set of clauses from the first set of content items by separating a content item into two or more clauses based on a presence of a connective punctuation mark between the two or more clauses.

13. The apparatus of claim 11 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

generate the set of clauses from the first set of content items by separating a content item into two or more clauses based on a presence of a connective word between the two or more clauses.

14. The apparatus of claim 11 , wherein obtaining the set of stop words comprising high-frequency words that occur in the second set of content items comprises:

ordering a set of words in the second set of content items by frequency; and

including a pre-specified number of most frequent words from the ordered set of words in the set of stop words.

15. The apparatus of claim 11 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

for each topic in the set of topics, automatically extract a set of attributes that provides a context for the topic from a subset of the content items containing the topic.

16. The apparatus of claim 15 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

obtain one or more sentiments associated with the topic; and

group the set of attributes by the one or more sentiments.

17. A system, comprising:

a topic-mining non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the system to:

obtain a set of clauses in a first set of content items comprising unstructured data;

obtain a set of stop words comprising high-frequency words that occur in a second set of content items; and

automatically extract a set of topics from the set of clauses by:

generating a set of n-grams from the set of clauses;

excluding a first n-gram from the set of n-grams when the first n-gram contains a word in the set of stop words in a pre-specified position of the first n-gram; and

applying a morphological filter to the set of n-grams to yield a subset of the set of n-grams, wherein:

the set of topics comprises the subset of the set of n-grams; and

the morphological filter is independent of words in the set of stop words;

wherein extracting the set of topics from the set of clauses further comprises:

excluding a second n-gram from the set of n-grams when the second n-gram contains a proportion of words in the set of stop words that exceeds a threshold; and

a management non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the system to display the set of topics to a user to improve understanding of the first set of content items by the user without requiring the user to manually analyze the first set of content items.

18. The system of claim 17 , wherein obtaining the set of stop words comprising high-frequency words that occur in the second set of content items comprises:

ordering a set of words in the second set of content items by frequency; and

including a pre-specified number of most frequent words from the ordered set of words in the set of stop words.

19. The system of claim 17 , wherein the topic-mining non-transitory computer-readable medium further comprises instructions that, when executed by one or more processors, cause the system to, for each of one or more topics in the set of topics:

automatically extract, from the set of clauses, a set of attributes that provides a context for the topic, wherein the set of attributes comprises a set of words in a vicinity of the topic.

20. The system of claim 19 , wherein the topic-mining non-transitory computer-readable medium further comprises instructions that, when executed by one or more processors, cause the system to, for each of the one or more topics:

obtain a refined set of attributes by removing from the set of attributes one or more high-frequency words that appear in the set of n-grams; and

use the refined set of attributes to obtain a set of sentiments associated with the topic.