System and methods thereof for mining web based user generated content for creation of term taxonomies
A system for creation of term taxonomies by mining web based user generated content according. The system includes a network interface enabling access to one or more data sources; a mining unit for collecting textual content from the one or more sources and generating phrases, the generated phrases include sentiment phrases and non-sentiment phrases; an analysis unit for generating at least associations between a non-sentiment phrase and a sentiment phrase based on the generated phrases, wherein an association between a non-sentiment phrase and at least one corresponding sentiment phrase is a taxonomy; and storing the taxonomies in a data warehouse storage connected to the network wherein responsive to a query the system provides a sentiment to a non-sentiment phrase provided in the query.
1. A system for creation of term taxonomies by mining web-based user generated content, comprising:
a network interface enabling access to one or more sources of information connected to a network;
a mining unit for collecting textual content from the one or more sources of information and generating phrases, the mining unit is further configured to identify in the generated phrases sentiment phrases and non-sentiment phrases based on a semantic meaning and context of each of the generated phrases in the collected textual content, each of the identified sentiment phrases includes one or more words describing a sentiment, the sentiment being any one of a positive sentiment, a neutral sentiment, and a negative sentiment, each of the positive sentiment, the neutral sentiment, and the negative sentiment is associated with a different score, wherein each of the generated sentiment phrases includes one or more words that do not describe a sentiment, wherein a sentiment phrase within a non-sentiment phrase is a non-sentiment phrase;
wherein the mining unit is further configured to perform a cleaning process on the identified sentiment phrases and non-sentiment phrases, wherein the cleaning process includes at least conversion of common misspelled phrases that appear in the textual content;
an analysis unit for generating a plurality of term taxonomies, wherein the analysis unit is configured to generate a term taxonomy of the plurality of term taxonomies by creating at least associations between a non-sentiment phrase of the identified non-sentiment phrases and a sentiment phrase of the identified sentiment phrases, wherein an association between the non-sentiment phrase and the at least one corresponding sentiment phrase is a term taxonomy, wherein the analysis unit is further configured to provide a trend report respective of a term taxonomy of the generated term taxonomies;
a non-transitory data warehouse storage medium connected to the network for storing the plurality of generated term taxonomies, wherein responsive to a query the system provides a sentiment of a non-sentiment phrase provided in the query; and
an alert unit for generating an alert responsive of the trend report.
2. The system of claim 1 , wherein the mining unit is connected to a phrase database containing identified non-sentiment phrases and sentiment phrases, wherein the mining unit further compares phrases in the textual content to phrases stored in the phrase database, wherein the phrase database further contains sentiment phrases having a corresponding score.
3. The system of claim 1 , wherein the score is any one of:
a negative score, a neutral score, and a positive score, wherein the negative score is a range of negative values and wherein the positive score is a range of positive values.
4. The system of claim 3 , wherein the score is initially assigned by a user of the system and adaptively adapted by a tuning mechanism.
5. The system of claim 4 , wherein the tuning mechanism further provides a feedback to at least one of: the mining unit and the analysis unit.
6. The system of claim 1 , wherein the mining unit computes a weighted sentiment score corresponding to a plurality of sentiment phrases collected for a respective non-sentiment phrase, wherein the weighted sentiment score is indicative of any one of the positive sentiment, the neutral sentiment, and the negative sentiment.
7. The system of claim 1 , wherein the analysis unit further correlates at least two non-sentiment phrases.
8. The system of claim 7 , wherein the analysis unit further compares a sentiment between the at least two non-sentiment phrases.
9. The system of claim 1 , wherein the alert unit is further configured to generates the alert responsive of at least one of: a trend detected from the trend report and a crossing of a threshold determined from the trend report.
10. The system of claim 1 , wherein each of the one or more sources of information is any one of: a social network, a blog, and a web page.
11. The system of claim 1 , wherein the common misspelled phrases are collected from a social network.
12. A method for generating term taxonomies between textual phrases, comprising:
crawling one or more sources of information by an agent operative on a computing device to collect textual content from the one or more data sources;
performing phrase extraction using the textual content to generate phrases;
identifying sentiment phrases and non-sentiment phrases from the generated phrases, wherein the identification is based on a semantic meaning and context of each of the generated phrases in the collected textual content, wherein each of the identified sentiment phrases includes one or more words describing a sentiment, the sentiment being any one of a positive sentiment, a neutral sentiment, and a negative sentiment, each of the positive sentiment, the neutral sentiment, and the negative sentiment is associated with a different score, wherein each of the generated sentiment phrases includes one or more words that do not describe a sentiment, wherein a sentiment phrase within a non-sentiment phrase is a non-sentiment phrase;
performing a cleaning process on the identified sentiment and non-sentiment phrases, wherein the cleaning process includes at least conversion of common misspelled phrases that appear in the textual content; associating a sentiment phrase with at least one non-sentiment phrase to create at least a term taxonomy; and
storing the term taxonomy of each of the at least one non-sentiment phrase having a corresponding sentiment phrase in a data warehouse storage;
generating a trend report respective of the term taxonomy of at least a sentiment phrase with respect of at least a non-sentiment phrase; and
generating an alert responsive of the trend report.
13. The method of claim 12 , wherein identifying the sentiment phrases and non-sentiment phrases further comprises:
comparing each of the generated phrases to sentiment phrases and non-sentiment phrases stored in a phrase database, wherein at least the sentiment phrase in the phrase database is associated with a score;
determining that a phrase is a sentiment phrase when a match is found between the phrase and one of the sentiment phrases stored in the phrase database; and
determining that a phrase is a non-sentiment phrase when a match is not found between the phrase and one of the sentiment phrases in the phrase database.
14. The method of claim 12 , wherein the score is any one of: a negative score, a neutral score, and a positive score, wherein the negative score is a range of negative values and wherein the positive score is a range of positive values.
15. The method of claim 14 , further comprising:
generating a weighted sentiment score corresponding to a plurality of sentiment phrases collected for a respective non-sentiment phrase.
16. The method of claim 12 , further comprising:
correlating the non-sentiment phrase with at least another non-sentiment phrase.
17. The method of claim 16 , wherein the correlated non-sentiment phrases are associated with the same sentiment phrase.
18. The method of claim 12 , further comprising:
receiving a feedback for the purpose of tuning at least one of: generation of phrases, generation of sentiment phrases, generation of non-sentiment phrases, correlation of non-sentiment phrases, and scores corresponding to sentiment phrases in the phrase database.
19. The method of claim 12 , wherein
generating the alert responsive of the trend report further comprising generating the alert responsive of at least one of: a trend detected from the trend report and a crossing of a threshold determined from the trend report.
20. The method of claim 12 , wherein each of the one or more sources of information is at least one of: a social network, a blog, and a web page.
21. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 12 .
22. The method of claim 12 , wherein the common misspelled phrases are collected from a social network.