System and method for determining sentiment expressed in documents

1. A computer-implemented method for determining sentiment expressed in a document, comprising:

receiving a document from a plurality of documents;

using at least one processor, identifying a sentence in the document that includes at least one sentiment signature within a predetermined distance of at least one keyword from a list of keywords, wherein the list of keywords is extracted from the plurality of documents and is filtered using a phase transition formula, wherein the phase transition formula is f/N x , where f is a frequency of the keyword in the plurality of documents, N is a number of documents that include the keyword, and x>1, and wherein the at least one sentiment signature corresponds to an expression of at least one sentiment in the sentence;

determining at least one category corresponding to the at least one keyword of the sentence, wherein the at least one category is included in a list of categories that is generated using the list of keywords;

determining at least one sentiment corresponding to the at least one category based on the at least one sentiment signature.

2. The computer-implemented method of claim 1 , wherein prior to identifying the sentence in the document that includes the at least one sentiment signature within the predetermined distance of the at least one keyword, the method further comprises extracting the list of keywords by:

extracting keywords from each document of the plurality of documents;

for each keyword,

calculating a frequency, f, of the keyword in the plurality of documents and a number of documents, N, that include the keyword;

using the phase transition formula to calculate the relevancy of the keyword based on the frequency of the keyword in the plurality of documents and the number of documents that include the keyword; and

adding the keyword to the list of keywords when the relevancy of the keyword exceeds a predetermined threshold.

3. The computer-implemented method of claim 1 , wherein x is 3.

4. The computer-implemented method of claim 1 , wherein prior to determining the at least one category corresponding to the at least one keyword of the sentence, the method further comprises generating the list of categories by:

identifying a first set of documents in the plurality of documents that include at least one keyword from the list of keywords;

identifying sets of keywords that are included in at least a predetermined number of documents in the first set of documents; and

adding the sets of keywords to the list of categories, wherein a respective category includes a respective set of keywords.

5. The computer-implemented method of claim 1 , wherein prior to determining the at least one category corresponding to the at least one keyword of the sentence, the method further comprises generating the list of categories by:

determining pairs of keywords in the list of keywords that are related to each other, wherein the pairs of keywords are unique pairs of keywords;

identifying sets of the pairs of the keywords in which each set includes at least one keyword that is common to all of the pairs of keywords in the set; and

until a predetermined termination condition is achieved, iteratively combining the set of the pairs of keywords in which each combined set includes at least one keyword that is common to all of the pairs of keywords in the combined set.

6. The computer-implemented method of claim 1 , wherein determining the at least one category corresponding to the at least one keyword of the sentence includes using a support vector machine to determine the at least one category corresponding to the at least one keyword of the sentence.

7. The computer-implemented method of claim 1 , wherein determining the at least one category corresponding to the at least one keyword of the sentence includes using a neural network to determine the at least one category corresponding to the at least one keyword of the sentence.

8. The computer-implemented method of claim 1 , wherein determining the at least one category corresponding to the at least one keyword of the sentence includes:

obtaining a plurality of category spectrums, a respective category spectrum including a frequency of occurrence of keywords in the list of keywords that corresponds to a respective category;

determining a category spectrum for the sentence based on the at least one keyword;

calculating dot products of the category spectrum for the sentence and each category spectrum in the plurality of category spectrums; and

determining the at least one category as a category corresponding to at least one dot product that exceeds a predetermined threshold.

9. The computer-implemented method of claim 8 , wherein prior to obtaining the plurality of category spectrums, the method further comprises for each category, determining a category spectrum for the category by:

obtaining a corpus of documents corresponding to the category;

extracting keywords from each document in the corpus of documents;

filtering the keywords using the phase transition formula to produce filtered keywords;

determining the frequency of occurrence of the filtered keywords in the corpus of documents; and

normalizing the frequency of occurrence of the filtered keywords to produce the category spectrum for the category.

10. The computer-implemented method of claim 1 , wherein prior to receiving the document from the plurality of documents, the method further comprises selecting the plurality of documents from a corpus of documents by:

for each document in the corpus of documents,

extracting n-grams from the document;

determining an n-gram spectrum for the document based on the extracted n-grams, wherein the n-gram spectrum indicates a frequency of occurrence of n-grams as a function of a size of n-grams;

determining whether the n-gram spectrum for the document conforms to a reference n-gram spectrum within a predetermined threshold, wherein the reference n-gram spectrum is defined by a predetermined equation;

adding the document to the plurality of documents when the n-gram spectrum for the document conforms to the reference n-gram spectrum within the predetermined threshold; and

discarding the document when the n-gram spectrum for the document does not conform to the reference n-gram spectrum within the predetermined threshold.

11. The computer-implemented method of claim 10 , wherein the predetermined equation is cx −a ·e −bx , wherein x is the size of the n-gram, and wherein a, b, and c are predetermined values that place a peak of the predetermined equation between an n-gram of size 2 and an n-gram size 3.

12. The computer-implemented method of claim 1 , wherein the at least one sentiment signature includes at least one word indicating that the expression of the at least one sentiment exists in the sentence.

13. The computer-implemented method of claim 1 , wherein the at least one sentiment is an expression of an opinion relating to at least one category.

14. The computer-implemented method of claim 1 , wherein the at least one category is associated with a product.

15. The computer-implemented method of claim 1 , wherein the at least one category is associated with a service.

16. A system to determine sentiment expressed in a document, comprising:

at least one processor;

memory; and

at least one program stored in the memory, the at least one program comprising instructions to:

receive a document from a plurality of documents;

identify a sentence in the document that includes at least one sentiment signature within a predetermined distance of at least one keyword from a list of keywords, wherein the list of keywords is extracted from the plurality of documents and is filtered using a phase transition formula, wherein the phase transition formula is f/N x , where f is a frequency of the keyword in the plurality of documents, N is a number of documents that include the keyword, and x>1, and wherein the at least one sentiment signature corresponds to an expression of at least one sentiment in the sentence;

determine at least one category corresponding to the at least one keyword of the sentence, wherein the at least one category is included in a list of categories that is generated using the list of keywords; and

determine at least one sentiment corresponding to the at least one category based on the at least one sentiment signature.

17. The system of claim 16 , wherein further comprising instructions to extract the list of keywords, wherein the instructions to extract the list of keywords from the document include instructions to:

extract keywords from each document of the plurality of documents;

for each keyword,

calculate a frequency, f, of the keyword in the plurality of documents and a number of documents, N, that include the keyword;

use the phase transition formula to calculate the relevancy of the keyword based on the frequency of the keyword in the plurality of documents and the number of documents that include the keyword; and

add the keyword to the list of keywords when the relevancy of the keyword exceeds a predetermined threshold.

18. A computer readable storage device storing at least one program configured for execution by a computer, the at least one program comprising instructions to:

receive a document from a plurality of documents;

identify a sentence in the document that includes at least one sentiment signature within a predetermined distance of at least one keyword from a list of keywords, wherein the list of keywords is extracted from the plurality of documents and is filtered using a phase transition formula, wherein the phase transition formula is f/N x , where f is a frequency of the keyword in the plurality of documents, N is a number of documents that include the keyword, and x>1, and wherein the at least one sentiment signature corresponds to an expression of at least one sentiment in the sentence;

determine at least one category corresponding to the at least one keyword of the sentence, wherein the at least one category is included in a list of categories that is generated using the list of keywords; and

determine at least one sentiment corresponding to the at least one category based on the at least one sentiment signature.