Realtime data stream cluster summarization and labeling system
A method is provided for automatically discovering topics in electronic posts, such as social media posts. The method includes receiving a corpus that includes a plurality of electronic posts. The method further includes identifying a plurality of candidate terms within the corpus and selecting, as a trimmed lexicon, a subset of the plurality of candidate terms using predefined criteria. The method further includes clustering at least a subset of the plurality of electronic posts according to a plurality of clusters using the lexicon to produce a plurality of statistical topic models. The method further includes storing information corresponding to the statistical topic models.
1. A method for generating topic labels from statistical topic models associated with a plurality of electronic communications, the method comprising:
at a computer system having real-time access to a data stream including the plurality of electronic communications, the computer system including a plurality of processors and memory storing programs for execution by the processors:
receiving a collection of topics derived from the plurality of electronic communications, associated topic word probabilities for a given topic in conjunction with a statistical topic model, and a set of documents associated with each topic;
truncating a document set to include documents having an aggregate topic word probability that meets truncation criteria;
determining, for each document in the truncated document set for the given topic, an aggregate topic word probability;
for one or more topic words in the truncated document set, identifying topic fragments including the one or more topic words and one or more non-stopwords, wherein identifying topic fragments includes iterating backwards and forwards from the one or more topic words and storing stopwords positioned relative to the one or more topic words until respective non-stopwords are identified; and
generating a topic label for the truncated document set including one or more of the identified topic fragments.
2. The method of claim 1 , wherein the statistical topic models are generated by a latent Dirichlet allocation (LDA) process.
3. The method of claim 1 , wherein the truncation criteria include a criterion that is met when the aggregate topic word probability for a document exceeds a truncation threshold.
4. The method of claim 1 , further comprising:
reweighting the probabilities for each topic word in the truncated document set for a given topic based on the frequency that the topic word appears across the collection of topics;
wherein determining, for each document in the truncated document set for the given topic, an aggregate topic word probability is based on the aggregate of the reweighted probabilities for each topic word for a given topic.
5. The method of claim 4 , wherein reweighting the probabilities for each topic word in the truncated document set comprises:
determining a total number of topics;
determining a global count of a word across all topics; and
multiplying a topic word probability for a given topic by the global count divided by the total number of topics.
6. A system for generating topic labels from statistical topic models associated with a plurality of electronic communications, the system including one or more processors and memory storing one or more programs to be executed by the one or more processors, the one or more programs including instructions for:
receiving a collection of topics derived from the plurality of electronic communications, associated topic word probabilities for a given topic in conjunction with a statistical topic model, and documents associated with each topic;
truncating a document set to include documents having an aggregate topic word probability that meets truncation criteria;
determining, for each document in the truncated document set for the given topic, an aggregate topic word probability;
for one or more topic words in the truncated document set, identifying topic fragments including the one or more topic words and one or more non-stopwords, wherein identifying topic fragments includes iterating backwards and forwards from the one or more topic words and storing stopwords positioned relative to the one or more topic words until respective non-stopwords are identified; and
generating a topic label for the truncated document set including one or more of the identified topic fragments.
7. The system of claim 6 , wherein the statistical topic models are generated by a latent Dirichlet allocation (LDA) process.
8. The system of claim 6 , wherein the truncation criteria include a criterion that is met when the aggregate topic word probability for a document exceeds a truncation threshold.
9. The system of claim 6 , further comprising:
reweighting the probabilities for each topic word in the truncated document set for a given topic based on the frequency that the topic word appears across the collection of topics;
wherein determining, for each document in the truncated document set for the given topic, an aggregate topic word probability is based on the aggregate of the reweighted probabilities for each topic word for a given topic.
10. The system of claim 9 , wherein reweighting the probabilities for each topic word in the truncated document set comprises:
determining a total number of topics;
determining a global count of a word across all topics; and
multiplying a topic word probability for a given topic by the global count divided by the total number of topics.
11. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system having real-time access to a data stream including the plurality of electronic communications, the one or more programs including instructions for:
receiving a collection of topics derived from the plurality of electronic communications, associated topic word probabilities for a given topic in conjunction with a statistical topic model, and documents associated with each topic;
truncating a document set to include documents having an aggregate topic word probability that meets truncation criteria;
determining, for each document in the truncated document set for the given topic, an aggregate topic word probability;
for one or more topic words in the truncated document set, identifying topic fragments including the one or more topic words and one or more non-stopwords, wherein identifying topic fragments includes iterating backwards and forwards from the one or more topic words and storing stopwords positioned relative to the one or more topic words until respective non-stopwords are identified; and
generating a topic label for the truncated document set including one or more of the identified topic fragments.
12. The non-transitory computer readable storage medium of claim 11 , wherein the statistical topic models are generated by a latent Dirichlet allocation (LDA) process.
13. The non-transitory computer readable storage medium of claim 11 , wherein the truncation criteria include a criterion that is met when the aggregate topic word probability for a document exceeds a truncation threshold.
14. The non-transitory computer readable storage medium of claim 11 , further comprising:
reweighting the probabilities for each topic word in the truncated document set for a given topic based on the frequency that the topic word appears across the collection of topics;
wherein determining, for each document in the truncated document set for the given topic, an aggregate topic word probability is based on the aggregate of the reweighted probabilities for each topic word for a given topic.
15. The non-transitory computer readable storage medium of claim 14 , wherein reweighting the probabilities for each topic word in the truncated document set comprises:
determining a total number of topics;
determining a global count of a word across all topics; and
multiplying a topic word probability for a given topic by the global count divided by the total number of topics.