System and method detecting hidden connections among phrases
A system and method for identifying hidden connections among non-sentiment phrases are presented. The method includes identifying all connections among a plurality of non-sentiment phrases based on at least one proximity rule; determining direct connections among the identified connections, wherein each direct connection meets a predetermined correlation; filtering out the determined direct connections from the identified connections to yield hidden connections among the identified connections; analyzing the hidden connections to identify a common phrase, wherein the common phrase is associated with at least two hidden connections; generating a new hidden connection among the plurality of non-sentiment phrases based on the common phrase; and associating a sentiment phrase with at least two non-sentiment phrases having a hidden connection, wherein the association is a term taxonomy.
1. A method for identifying hidden connections among non-sentiment phrases, comprising:
receiving textual content from a data source, the textual content comprising a plurality of non-sentiment phrases, wherein a proximity of mention between non-sentiment phrases potentially provides an indication that there is a likelihood of impact from one of the phrases to another, said impact constitutes a hidden-connection between the phrases;
identifying all connections among the plurality of non-sentiment phrases based on at least one proximity rule;
determining direct connections among the identified connections, wherein each direct connection meets a predetermined correlation;
filtering out the determined direct connections from the identified connections to yield hidden connections among the identified connections;
analyzing the hidden connections to identify a common phrase, wherein the common phrase is associated with at least two hidden connections, each hidden connection of the at least two hidden connections associated with at least two terms;
generating a new hidden connection associated with at least two terms, each of the at least two terms comprised in a hidden connection, among the plurality of non-sentiment phrases based on the common phrase;
associating a sentiment phrase with at least two non-sentiment phrases having a hidden connection, wherein the association is a term taxonomy; and
providing content for delivery based on sentiment phrase association.
2. The method of claim 1 , further comprising: extracting a plurality of phrases from textual content; and identifying a plurality of sentiment phrases and the plurality of non-sentiment phrases in the plurality of phrases, wherein the associated sentiment phrase is from the plurality of sentiment phrases.
3. The method of claim 2 , wherein identifying a plurality of sentiment phrases and the plurality of non-sentiment phrases in the plurality of phrases further comprises:
comparing each extracted phrase to a plurality of predetermined sentiment phrases and non-sentiment phrases; and
for each extracted phrase, determining whether the extracted phrase is a sentiment phrase or a non-sentiment phrase, wherein the extracted phrase is determined to be a sentiment phrase upon the extracted phrase matching one of the predetermined sentiment phrases, and wherein further the extracted phrase is determined to be a non-sentiment phrase upon the extracted phrase matching one of the predetermined non-sentiment phrases.
4. The method of claim 3 , further comprising: assigning a sentiment score to each determined sentiment phrase, wherein the sentiment score represents a degree of any of: positivity, neutrality, and negativity.
5. The method of claim 4 , further comprising: continuously monitoring feedback related to the determined sentiment phrases to identify a change in degree; and assigning a new sentiment score based on the change in degree.
6. The method of claim 3 , wherein each sentiment score is weighted respective of each other sentiment score.
7. The method of claim 2 , wherein identifying a plurality of sentiment phrases and the plurality of non-sentiment phrases in the plurality of phrases further comprises any of: replacing slang phrases with corresponding standard language phrases, removing conjunctions from phrases, removing unknown words, removing repeated words, and removing irrelevant words.
8. The method of claim 1 , wherein the proximity rule includes at least one of: a number of mentions of at least two non-sentiment phrases in a web page, a number of mentions of at least two non-sentiment phrases in different web pages, a number of mentions of at least two non-sentiment phrases in a piece of collected textual content, and a number of web pages within a web site between at least two mentioned non-sentiment phrases.
9. The method of claim 1 , wherein each hidden connection is any of: a first degree separation between at least two non-sentiment phrases, and a first degree separation between at least two different non-sentiment phrases that are associated with a common non-sentiment phrase.
10. The method of claim 1 , wherein at least two non-sentiment phrases have a connection when they meet the at least one proximity rule.
11. The method of claim 1 , wherein each hidden connection is between at least two non-sentiment phrases.
12. The method of claim 1 , wherein generating a new hidden connection further comprises: removing the identified common phrase.
13. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute a process for improving content provisioning by identifying hidden connections among non-sentiment phrases in such document, the process comprising:
receiving textual content from a data source, the textual content comprising a plurality of non-sentiment phrases, wherein a proximity of mention between non-sentiment phrases potentially provides an indication that there is a likelihood of impact from one of the phrases to another, said impact constitutes a hidden-connection between the phrases;
identifying all connections among the plurality of non-sentiment phrases based on at least one proximity rule;
determining direct connections among the identified connections, wherein each direct connection meets a predetermined correlation;
filtering out the determined direct connections from the identified connections to yield hidden connections among the identified connections;
analyzing the hidden connections to identify a common phrase, wherein the common phrase is associated with at least two hidden connections, each hidden connection of the at least two hidden connections associated with at least two terms;
generating a new hidden connection associated with at least two terms, each of the at least two terms comprised in a hidden connection, among the plurality of non-sentiment phrases based on the common phrase;
associating a sentiment phrase with at least two non-sentiment phrases having a hidden connection, wherein the association is a term taxonomy; and
providing content for delivery based on sentiment phrase association.
14. A system for improving content provisioning by identifying hidden connections among non-sentiment phrases in such document, comprising:
a processor; and
a memory, the memory containing instructions that, when executed by the processor configure the system to:
receive textual content from a data source, the textual content comprising a plurality of non-sentiment phrases, wherein a proximity of mention between non-sentiment phrases potentially provides an indication that there is a likelihood of impact from one of the phrases to another, said impact constitutes a hidden-connection between the phrases;
identify all connections among the plurality of non-sentiment phrases based on at least one proximity rule;
determine direct connections among the identified connections, wherein each direct connection meets a predetermined correlation;
filter out the determined direct connections from the identified connections to yield hidden connections among the identified connections;
analyze the hidden connections to identify a common phrase, wherein the common phrase is associated with at least two hidden connections, each hidden connection of the at least two hidden connections associated with at least two terms;
generate a new hidden connection associated with at least two terms, each of the at least two terms comprised in a hidden connection, among the plurality of non-sentiment phrases based on the common phrase;
associate a sentiment phrase with at least two non-sentiment phrases having a hidden connection, wherein the association is a term taxonomy; and
provide content for delivery based on sentiment phrase association.
15. The system of claim 14 , wherein the system is further configured to:
extract a plurality of phrases from textual content; and
identify a plurality of sentiment phrases and the plurality of non-sentiment phrases in the plurality of phrases, wherein the associated sentiment phrase is from the plurality of sentiment phrases.
16. The system of claim 15 , wherein identifying a plurality of sentiment phrases and the plurality of non-sentiment phrases in the plurality of phrases further comprises:
compare each extracted phrase to a plurality of predetermined sentiment phrases and non-sentiment phrases; and
for each extracted phrase, determine whether the extracted phrase is a sentiment phrase or a non-sentiment phrase, wherein the extracted phrase is determined to be a sentiment phrase if the extracted phrase matched one of the predetermined sentiment phrases, wherein further the extracted phrase is determined to be a non-sentiment phrase if the extracted phrase matched one of the predetermined non-sentiment phrases.
17. The system of claim 16 , wherein the system is further configured to: assign a sentiment score to each determined sentiment phrase, wherein the sentiment score represents a degree of any of: positivity, neutrality, and negativity.
18. The system of claim 15 , wherein the system is further configured to:
continuously monitor feedback related to the determined sentiment phrases to identify a change in degree; and
assign a new sentiment score based on the change in degree.
19. The system of claim 16 , wherein each sentiment score is weighted respective of each other sentiment score.
20. The system of claim 15 , the system is further configured to perform any of: replace slang phrases with corresponding standard language phrases, remove conjunctions from phrases, remove unknown words, remove repeated words, and remove irrelevant words.
21. The system of claim 14 , wherein the proximity rule includes at least one of: a number of mentions of at least two non-sentiment phrases in a web page, a number of mentions of at least two non-sentiment phrases in different web pages, a number of mentions of at least two non-sentiment phrases in a piece of collected textual content, and a number of web pages within a web site between at least two mentioned non-sentiment phrases.
22. The system of claim 14 , wherein each hidden connection is any of: a first degree separation between at least two non-sentiment phrases, and a first degree separation between at least two different non-sentiment phrases that are associated with a common non-sentiment phrase.
23. The system of claim 14 , wherein at least two non-sentiment phrases have a connection when they meet the at least one proximity rule.
24. The system of claim 14 , wherein each hidden connection is between at least two non-sentiment phrases.
25. The system of claim 14 , wherein the system is further configured to: remove the identified common phrase.