Automatic topic discovery in streams of unstructured data

1. A method of automatically identifying high-value electronic posts from electronic streams of unstructured data, in real-time, using statistical topic models, comprising:

in a computer including one or more processors and a memory storing instructions for execution by the one or more processors:

receiving a corpus that includes a plurality of electronic posts;

identifying, within the corpus, a plurality of candidate terms, at least one of the candidate terms comprising a first word and a second word, the first word and the second word being separated by one or more third words, wherein the at least one of the candidate terms includes a numerical value representing a count of the third words separating the first word and the second word;

selecting, as a trimmed lexicon, a subset of the plurality of candidate terms according to predefined criteria, wherein the predefined criteria are based on (i) a frequency of each word that comprises a respective candidate term appearing in the corpus, wherein each word that comprises a respective candidate term includes a respective first word and a respective second word but no respective third words, and (ii) the numerical value representing the count of third words that separate the first word and the second word that comprises a respective candidate term, wherein selecting, as the trimmed lexicon, the subset of the plurality of candidate terms according to predefined criteria includes weighting respective combinations of first words and second words by the respective numerical values representing the respective counts of third words, wherein a greater respective count of third words weighs against selection of a respective first and second word;

clustering at least a subset of the plurality of electronic posts according to a plurality of clusters using the trimmed lexicon to produce a plurality of statistical topic models;

storing information corresponding to the statistical topic model;

receiving a subsequent electronic post, wherein the subsequent electronic post is received after the corpus that includes the plurality of electronic posts; and

clustering the subsequent electronic post to at least one of the statistical topic models in real-time based on the information corresponding to the statistical topic model.

2. The method of claim 1 , wherein each statistical topic model corresponds to a topic that is represented as a probability distribution over the trimmed lexicon.

3. The method of claim 1 , wherein the plurality of electronic posts includes a plurality of social media posts.

4. The method of claim 1 , wherein selecting, as the trimmed lexicon, the subset of the plurality of candidate terms according to predefined criteria includes selecting a predefined number or predefined fraction of the candidate terms.

5. The method of claim 1 , wherein clustering the subset of the plurality of electronic posts includes assigning each electronic post in the subset of the plurality of electronic posts a probability corresponding to each cluster in the plurality of clusters.

6. The method of claim 1 , wherein receiving the corpus includes receiving a data stream of electronic posts and filtering the data stream of electronic posts according to one or more topic filters, wherein the plurality of electronic posts include respective electronic posts that are accepted by the one more topic filters.

7. The method of claim 1 , wherein selecting, as the trimmed lexicon, the subset of the plurality of candidate terms according to predefined criteria further includes weighting each respective candidate term according to a frequency with which the candidate term appears in the corpus divided by a total number of candidate terms that appear in the corpus.

8. The method of claim 1 , wherein selecting, as the trimmed lexicon, the subset of the plurality of candidate terms according to predefined criteria further includes weighting each respective candidate term according to a fraction of electronic posts in which the respective candidate term appears.

9. The method of claim 1 , wherein selecting, as the trimmed lexicon, the subset of the plurality of candidate terms according to predefined criteria further includes weighting the candidate terms according to at least one of: a term frequency-inverse document frequency metric, an entropy metric, and a point-wise mutual information metric.

10. The method of claim 1 , further including, prior to identifying, within the corpus, the plurality of candidate terms, normalizing the plurality of electronic posts by performing one or more of the following operations on content within the plurality of electronic posts:

stop term removal;

spelling correction;

synonym mapping;

token downcasing; and

duplicate post removal.

11. The method of claim 1 , wherein each cluster of the plurality of clusters represents a respective topic.

12. The method of claim 1 , further including, automatically, without user intervention, labeling each respective cluster using a respective term of the trimmed lexicon that meets one or more prevalence criteria.

13. The method of claim 1 , further including, after receiving the plurality of electronic posts:

indexing the plurality of electronic posts; and

storing the plurality of electronic posts.

14. The method of claim 1 , wherein clustering the subset of the plurality of electronic posts comprises performing latent Dirichlet allocation (LDA).

15. The method of claim 1 , further including, generating a statistical model including assigning each term in the trimmed lexicon a probability corresponding to each cluster in the plurality of clusters.

16. The method of claim 15 , further including:

receiving a second plurality of electronic posts; and

clustering the second plurality of electronic posts according to the statistical model.

17. A server system configured to automatically identify high-value electronic posts from electronic streams of unstructured data, in real-time, using statistical topic models, comprising one or more processors and memory, the memory including a non-transitory computer readable storage medium storing a set of instructions that cause the one or more processors to:

receive a corpus that includes a plurality of electronic posts;

identify, within the corpus, a plurality of candidate terms, at least one of the candidate terms comprising a first word and a second word, the first word and the second word being separated by one or more third words, wherein the at least one of the candidate terms includes a numerical value representing a count of the third words separating the first word and the second word;

select, as a trimmed lexicon, a subset of the plurality of candidate terms according to predefined criteria, wherein the predefined criteria are based on (i) a frequency of each word that comprises a respective candidate term appearing in the corpus, wherein each word that comprises a respective candidate term includes a respective first word and a respective second word but no respective third words, and (ii) the numerical value representing the count of third words that separate the first word and the second word that comprises a respective candidate term, wherein selecting, as the trimmed lexicon, the subset of the plurality of candidate terms according to predefined criteria includes weighting respective combinations of first words and second words by the respective numerical values representing the respective counts of third words, wherein a greater respective count of third words weighs against selection of a respective first and second word;

cluster at least a subset of the plurality of electronic posts according to a plurality of clusters using the lexicon to produce a plurality of statistical topic models;

store information corresponding to the statistical topic models;

receive a subsequent electronic post, wherein the subsequent electronic post is received after the corpus that includes the plurality of electronic posts; and

cluster the subsequent electronic post to at least one of the statistical topic models in real-time based on the information corresponding to the statistical topic model.

18. A non-transitory computer readable storage medium storing a set of instructions, which when executed by a server system with one or more processors cause the one or more processors to:

receive a corpus that includes a plurality of electronic posts;

identify, within the corpus, a plurality of candidate terms, at least one of the candidate terms comprising a first word and a second word, the first word and the second word being separated by one or more third words, wherein the at least one of the candidate terms includes a numerical value representing a count of the third words separating the first word and the second word;

select, as a trimmed lexicon, a subset of the plurality of candidate terms according to predefined criteria, wherein the predefined criteria are based on (i) a frequency of each word that comprises a respective candidate term appearing in the corpus, wherein each word that comprises a respective candidate term includes a respective first word and a respective second word but no respective third words, and (ii) the numerical value representing the count of third words that separate the first word and the second word that comprises a respective candidate term, wherein selecting, as the trimmed lexicon, the subset of the plurality of candidate terms according to predefined criteria includes weighting respective combinations of first words and second words by the respective numerical values representing the respective counts of third words, wherein a greater respective count of third words weighs against selection of a respective first and second word;

cluster at least a subset of the plurality of electronic posts according to a plurality of clusters using the lexicon to produce a plurality of statistical topic models;

store information corresponding to the statistical topic models;

receive a subsequent electronic post, wherein the subsequent electronic post is received after the corpus that includes the plurality of electronic posts; and

cluster the subsequent electronic post to at least one of the statistical topic models in real-time based on the information corresponding to the statistical topic model.