Sentiment analysis from social media content
The sentiment engine includes a sentiment module configured to gather opinions or determine sentiment expressed in documents, a crawling module configured to crawl servers to obtain at least a subset of the documents or opinions from social media websites, a keyword module configured to extract keywords from documents, a filtering module configured to filter keywords and documents, and a classification module configured to classify documents, sentences, and/or keywords, a polarity prediction module configured to predict the polarity of a sentiment sentence, and a social media net promoter score (SNPS) configured to calculate a loyalty metric of users from social media websites. The functionality of these modules may be combined with one another or in addition to other modules.
1. A computer-implemented method for sentiment analysis from social media content, comprising:
performing by at least one processor the steps of:
crawling a plurality of websites to obtain metadata from social media content;
extracting the metadata from the social media content by identifying at least one sentiment-bearing keyword and a polarity thereof in a sentence from the social media content;
determining at least one category corresponding to the at least one sentiment-bearing keyword of the sentence; and
determining at least one sentiment corresponding to the at least one category based on the at least one sentiment-bearing keyword,
wherein,
the method comprises the step of extracting a list of keywords from a plurality of documents, the step of extracting the list of keywords including:
(a) extracting keywords from each document of the plurality of documents, and
for each keyword,
(b) for each keyword,
calculating a frequency, f, of the keyword in the plurality of documents and a number of documents, N, that include the keyword,
using a phase transition formula to calculate a relevancy of the keyword based on the frequency of the keyword in the plurality of documents and the number of documents that include the keyword, and
adding the keyword to the list of keywords when the relevancy of the keyword exceeds a predetermined threshold.
2. The method of claim 1 , wherein the step of extracting includes filtering the at least one sentiment-bearing keyword with sentiment eliminators.
3. The method claim 1 , wherein the step of extracting includes filtering the at least one sentiment-bearing keyword by associating with a sentiment score.
4. The method of claim 1 , wherein the phase transition formula is:
f
N
x
,
wherein x≧1.
5. The method of claim 1 , wherein prior to the step of determining at least one category, the method further comprises the step of generating a list of categories by:
determining pairs of keywords in the list of keywords that are related to each other, the pairs of keywords being unique pairs of keywords,
identifying sets of the pairs of the keywords in which each set includes at least one keyword that is common to all of the pairs of keywords in the set, and
until a predetermined termination condition is achieved, iteratively combining the sets of the pairs of keywords in which each combined set includes at least one keyword that is common to all of the pairs of keywords in the combined set.
6. The method of claim 1 , wherein the step of determining at least one category includes using a neural network to determine the at least one category corresponding to the at least one sentiment-bearing keyword of the sentence.
7. The method of claim 1 , wherein the at least one sentiment is an expression of an opinion relating to the at least one category.
8. The method of claim 1 , wherein the at least one category is associated with a product.
9. The method of claim 1 , wherein the at least one category is associated with a service.
10. A computer-implemented method for sentiment analysis from social media content, comprising:
performing by at least one processor the steps of:
crawling a plurality of websites to obtain metadata from social media content;
extracting the metadata from the social media content by identifying at least one sentiment-bearing keyword and a polarity thereof in a sentence from the social media content;
determining at least one category corresponding to the at least one sentiment-bearing keyword of the sentence; and
determining at least one sentiment corresponding to the at least one category based on the at least one sentiment-bearing keyword,
wherein,
the step of determining at least one category includes:
(a) obtaining a plurality of category spectrums, a respective category spectrum including a frequency of occurrence of keywords in a list of keywords that corresponds to a respective category,
(b) determining a category spectrum for the sentence based on at least one keyword,
(c) calculating dot products of the category spectrum for the sentence and each category spectrum in the plurality of category spectrums, and
(d) determining the at least one category as a category corresponding to at least one dot product that exceeds a predetermined threshold.
11. The method of claim 10 , wherein prior to obtaining the plurality of category spectrums, the method further comprises for each category, determining a category spectrum for the category by:
obtaining a corpus of documents corresponding to the category,
extracting keywords from each document in the corpus of documents,
filtering the keywords using a phase transition formula to produce filtered keywords,
determining a frequency of occurrence of the filtered keywords in the corpus of documents, and
normalizing the frequency of occurrence of the filtered keywords to produce the category spectrum for the category.
12. A system comprising:
at least one processor;
memory; and
at least one program stored in the memory and executable by the at least one processor, the at least one program comprising instructions to:
crawl a plurality of websites to obtain metadata from social media content,
extract the metadata from the social media content by identifying at least one sentiment-bearing keyword and a polarity thereof in a sentence from the social media content,
determine at least one category corresponding to the at least one sentiment-bearing keyword of the sentence, and
determine at least one sentiment corresponding to the at least one category based on the at least one sentiment-bearing keyword,
wherein,
the instructions further include instructions to extract a list of keywords from a plurality of documents, the instructions to extract the list of keywords including instructions to:
(a) extract keywords from each document of the plurality of documents, and
(b) for each keyword,
calculate a frequency, f, of the keyword in the plurality of documents and a number of documents, N, that include the keyword,
use a phase transition formula to calculate a relevancy of the keyword based on the frequency of the keyword in the plurality of documents and the number of documents that include the keyword, and
add the keyword to the list of keywords when the relevancy of the keyword exceeds a predetermined threshold.
13. The system of claim 12 , wherein the phase transition formula is:
f
N
x
,
wherein x≧1.
14. A non-transitory computer readable storage medium storing at least one program configured for execution by a computer, the at least one program comprising instructions to:
crawl a plurality of websites to obtain metadata from social media content;
extract the metadata from the social media content by identifying at least one sentiment-bearing keyword and a polarity thereof in a sentence from the social media content;
determine at least one category corresponding to the at least one sentiment-bearing keyword of the sentence; and
determine at least one sentiment corresponding to the at least one category based on the at least one sentiment-bearing keyword,
wherein,
the instructions further include instructions to extract a list of keywords from a plurality of documents, the instructions to extract the list of keywords including instructions to:
(a) extract keywords from each document of the plurality of documents, and
(b) for each keyword,
calculate a frequency, f, of the keyword in the plurality of documents and a number of documents, N, that include the keyword,
use a phase transition formula to calculate a relevancy of the keyword based on the frequency of the keyword in the plurality of documents and the number of documents that include the keyword, and
add the keyword to the list of keywords when the relevancy of the keyword exceeds a predetermined threshold.
15. A system, comprising:
at least one processor;
memory; and
at least one program stored in the memory and executable by the at least one processor, the at least one program comprising instructions to:
crawl a plurality of websites to obtain metadata from social media content,
extract the metadata from the social media content by identifying at least one sentiment-bearing keyword and a polarity thereof in a sentence from the social media content,
determine at least one category corresponding to the at least one sentiment-bearing keyword of the sentence, and
determine at least one sentiment corresponding to the at least one category based on the at least one sentiment-bearing keyword,
wherein,
the instructions to determine at least one category include instructions to:
(a) obtain a plurality of category spectrums, a respective category spectrum including a frequency of occurrence of keywords in a list of keywords that corresponds to a respective category,
(b) determine a category spectrum for the sentence based on at least one keyword,
(c) calculate dot products of the category spectrum for the sentence and each category spectrum in the plurality of category spectrums, and
(d) determine the at least one category as a category corresponding to at least one dot product that exceeds a predetermined threshold.
16. A non-transitory computer readable storage medium storing at least one program configured for execution by a computer, the at least one program comprising instructions to:
crawl a plurality of websites to obtain metadata from social media content;
extract the metadata from the social media content by identifying at least one sentiment-bearing keyword and a polarity thereof in a sentence from the social media content;
determine at least one category corresponding to the at least one sentiment-bearing keyword of the sentence; and
determine at least one sentiment corresponding to the at least one category based on the at least one sentiment-bearing keyword,
wherein,
the instructions to determine at least one category include instructions to:
(a) obtain a plurality of category spectrums, a respective category spectrum including a frequency of occurrence of keywords in a list of keywords that corresponds to a respective category,
(b) determine a category spectrum for the sentence based on at least one keyword,
(c) calculate dot products of the category spectrum for the sentence and each category spectrum in the plurality of category spectrums, and
(d) determine the at least one category as a category corresponding to at least one dot product that exceeds a predetermined threshold.