TOPIC MINING USING NATURAL LANGUAGE PROCESSING TECHNIQUES
The disclosed embodiments provide a method, system and apparatus for processing data. During operation, the system obtains a set of content items containing unstructured data. Next, the system obtains a set of part-of-speech (POS) tags for lexical items in the set of content items. The system then uses a computer to match the POS tags to one or more POS tagging patterns to obtain a set of candidate topics for the set of content items and extract a set of topics for the set of content items from the set of candidate topics.
1 . A computer-implemented method for processing data, comprising:
obtaining a set of content items comprising unstructured data;
obtaining a set of part-of-speech (POS) tags for lexical items in the set of content items; and
using a computer to:
match the POS tags to one or more POS tagging patterns to obtain a set of candidate topics for the set of content items; and
extract a set of topics for the set of content items from the set of candidate topics.
2 . The computer-implemented method of claim 1 , further comprising:
cleaning the set of candidate topics prior to extracting the set of topics from the candidate topics.
3 . The computer-implemented method of claim 2 , wherein cleaning the set of candidate topics comprises at least one of:
performing stemming of the set of candidate topics;
removing stop words from the set of candidate topics;
merging synonyms in the set of candidate topics; and
merging semantically related lexical items in the set of candidate topics.
4 . The computer-implemented method of claim 3 , wherein the stop words and the synonyms are associated with use of an online professional network.
5 . The computer-implemented method of claim 1 , wherein the one or more POS tagging patterns comprise:
a recursive noun phrase;
a noun phrase followed by a verb phrase; and
the verb phrase followed by the noun phrase.
6 . The computer-implemented method of claim 1 , wherein extracting the set of topics from the set of candidate topics comprises:
filtering the candidate topics by a metric associated with the candidate topics.
7 . The computer-implemented method of claim 6 , wherein the metric is at least one of:
a term frequency;
a document frequency; and
an inverse document frequency.
8 . The computer-implemented method of claim 1 , wherein the set of content items comprises at least one of:
a customer survey;
a complaint;
a review;
a group discussion; and
social media content.
9 . A system for processing data, comprising:
a tagging apparatus configured to:
obtain a set of content items comprising unstructured data; and
obtain a set of part-of-speech (POS) tags for lexical items in the set of content items;
a matching apparatus configured to match the POS tags to one or more POS tagging patterns to obtain a set of candidate topics for the set of content items; and
an extraction apparatus configured to extract a set of topics for the set of content items from the set of candidate topics.
10 . The system of claim 9 , further comprising:
a cleaning apparatus configured to clean the set of candidate topics prior to extracting the set of topics from the candidate topics.
11 . The system of claim 10 , wherein cleaning the set of candidate topics comprises at least one of:
performing stemming of the set of candidate topics;
removing stop words from the set of candidate topics;
merging synonyms in the set of candidate topics; and
merging semantically related lexical items in the set of candidate topics.
12 . The system of claim 9 , wherein the one or more POS tagging patterns comprise:
a recursive noun phrase;
a noun phrase followed by a verb phrase; and
the verb phrase followed by the noun phrase.
13 . The system of claim 9 , wherein extracting the set of topics from the set of candidate topics comprises:
filtering the candidate topics by a metric associated with the candidate topics.
14 . The system of claim 9 , wherein the set of content items comprises at least one of:
a customer survey;
a complaint;
a review;
a group discussion; and
social media content.
15 . An apparatus, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
obtain a set of content items comprising unstructured data;
obtain a set of part-of-speech (POS) tags for lexical items in the set of content items;
match the POS tags to one or more POS tagging patterns to obtain a set of candidate topics for the set of content items; and
extract a set of topics for the set of content items from the set of candidate topics.
16 . The apparatus of claim 15 , wherein the instructions further cause the apparatus to:
clean the set of candidate topics prior to extracting the set of topics from the candidate topics.
17 . The apparatus of claim 16 , wherein cleaning the set of candidate topics comprises at least one of:
performing stemming of the set of candidate topics;
removing stop words from the set of candidate topics;
merging synonyms in the set of candidate topics; and
merging semantically related lexical items in the set of candidate topics.
18 . The apparatus of claim 15 , wherein the one or more POS tagging patterns comprise:
a recursive noun phrase;
a noun phrase followed by a verb phrase; and
the verb phrase followed by the noun phrase.
19 . The apparatus of claim 15 , wherein extracting the set of topics from the set of candidate topics comprises:
filtering the candidate topics by a metric associated with the candidate topics.
20 . The apparatus of claim 15 , wherein the set of content items comprises at least one of:
a customer survey;
a complaint;
a review;
a group discussion; and
social media content.