Realtime data stream cluster summarization and labeling system
A method is provided for automatically discovering topics in electronic posts, such as social media posts. The method includes receiving a corpus that includes a plurality of electronic posts. The method further includes identifying a plurality of candidate terms within the corpus and selecting, as a trimmed lexicon, a subset of the plurality of candidate terms using predefined criteria. The method further includes clustering at least a subset of the plurality of electronic posts according to a plurality of clusters using the lexicon to produce a plurality of statistical topic models. The method further includes storing information corresponding to the statistical topic models.
1. A method for generating topic labels from statistical topic models associated with a plurality of electronic communications, the method comprising:
at a computer system having real-time access to a data stream including the plurality of electronic communications, the computer system including a plurality of processors and memory storing programs for execution by the processors:
receiving a collection of topics derived from the plurality of electronic communications, associated topic word probabilities for a given topic in conjunction with a statistical topic model, and documents associated with each topic;
truncating a document set to include documents having an aggregate topic word probability that meets truncation criteria;
reweighting the probabilities for each topic word in the truncated document set for a given topic based on the frequency that the topic word appears across the collection of topics;
determining, for each document in the truncated document set for the given topic, an aggregate topic word probability;
for one or more topic words in the truncated document set:
iterating backwards from the one or more topic words and storing stopwords positioned relative to the one or more topic words until respective first non-stopwords are identified;
iterating forwards from the one or more topic words and storing stopwords positioned relative to the one or more topic words until respective second non-stopwords are identified; and
identifying topic fragments based on the one or more topic words and the respective first and second non-stopwords; and
generating a topic label for the truncated document set based on the identified topic fragments.
2. The method of claim 1 , wherein the statistical topic models are generated by a latent Dirichlet allocation (LDA) process.
3. The method of claim 1 , wherein truncation criteria includes a criterion that is met when the aggregate topic word probability for a document exceeds a truncation threshold.
4. The method of claim 1 , wherein determining, for each document in the truncated document set for the given topic, an aggregate topic word probability is based on the aggregate of the reweighted probabilities for each topic word for a given topic.
5. The method of claim 1 , wherein reweighting the probabilities for each topic word in the truncated document set comprises:
determining a total number of topics;
determining a global count of a word across all topics; and
multiplying a topic word probability for a given topic by the global count divided by the total number of topics.
6. A system for generating topic labels from statistical topic models associated with a plurality of electronic communications, the system including:
one or more processors; and
physical memory storing one or more programs to be executed by the one or more processors, the one or more programs including instructions for:
receiving a collection of topics derived from the plurality of electronic communications, associated topic word probabilities for a given topic in conjunction with a statistical topic model, and documents associated with each topic;
truncating a document set to include documents having an aggregate topic word probability that meets truncation criteria;
reweighting the probabilities for each topic word in the truncated document set for a given topic based on the frequency that the topic word appears across the collection of topics;
determining, for each document in the truncated document set for the given topic, an aggregate topic word probability;
for one or more topic words in the truncated document set:
iterating backwards from one or more topic words and storing stopwords positioned relative to the one or more topic words until respective first non-stopwords are identified;
iterating forwards from the one or more topic words and storing stopwords positioned relative to the one or more topic words until respective second non-stopwords are identified; and
identifying topic fragments based on the one or more topic words and the respective first and second non-stopwords; and
generating a topic label for the truncated document set based on the identified topic fragments.
7. The system of claim 6 , wherein the statistical topic models are generated by a latent Dirichlet allocation (LDA) process.
8. The system of claim 6 , wherein truncation criteria includes a criterion that is met when the aggregate topic word probability for a document exceeds a truncation threshold.
9. The system of claim 6 , wherein determining, for each document in the truncated document set for the given topic, an aggregate topic word probability is based on the aggregate of the reweighted probabilities for each topic word for a given topic.
10. The system of claim 6 , wherein the instructions for reweighting the probabilities for each topic word in the truncated document set comprise instructions for:
determining a total number of topics;
determining a global count of a word across all topics; and
multiplying a topic word probability for a given topic by the global count divided by the total number of topics.
11. A non-transitory computer readable storage medium storing one or more programs, which when executed by a computer system having real-time access to a data stream including a plurality of electronic communications, causes the computer system to perform the steps of:
receiving a collection of topics derived from the plurality of electronic communications, associated topic word probabilities for a given topic in conjunction with a statistical topic model, and documents associated with each topic;
truncating a document set to include documents having an aggregate topic word probability that meets truncation criteria;
reweighting the probabilities for each topic word in the truncated document set for a given topic based on the frequency that the topic word appears across the collection of topics;
determining, for each document in the truncated document set for the given topic, an aggregate topic word probability;
for one or more topic words in the truncated document set:
iterating backwards from the one or more topic words and storing stopwords positioned relative to the one or more topic words until respective first non-stopwords are identified;
iterating forwards from the one or more topic words and storing stopwords positioned relative to the one or more topic words until respective second non-stopwords are identified; and
identifying topic fragments based on the one or more topic words and the respective first and second non-stopwords; and
generating a topic label for the truncated document set based on the identified topic fragments.
12. The non-transitory computer readable storage medium of claim 11 , wherein the statistical topic models are generated by a latent Dirichlet allocation (LDA) process.
13. The non-transitory computer readable storage medium of claim 11 , wherein truncation criteria includes a criterion that is met when the aggregate topic word probability for a document exceeds a truncation threshold.
14. The non-transitory computer readable storage medium of claim 11 , wherein determining, for each document in the truncated document set for the given topic, an aggregate topic word probability is based on the aggregate of the reweighted probabilities for each topic word for a given topic.
15. The non-transitory computer readable storage medium of claim 11 , wherein the instructions for reweighting the probabilities for each topic word in the truncated document set comprise instructions for:
determining a total number of topics;
determining a global count of a word across all topics; and
multiplying a topic word probability for a given topic by the global count divided by the total number of topics.