Real-time event detection on social data streams

1. A method for event detection on social data streams, the method comprising:

identifying a plurality of trending entities from messages posted to a messaging platform, wherein a trending entity is an entity mentioned by users in the messages posted to the messaging platform at a rate higher than other entities;

executing one or more similarity-based clustering operations on the plurality of trending entities to detect a first cluster group of trending entities over a first period of time and a second cluster group of trending entities over a second period of time;

determining that the second cluster group is related to the first cluster group;

generating a cluster chain by linking the second cluster group with the first cluster group, the cluster chain representing an event over the first and second periods of time; and

storing the cluster chain in a memory device on the messaging platform.

2. The method of claim 1 , further comprising:

transmitting digital data to a client application of a user device to render information about the event in a user interface of the client application, the information identifying a first trending entity from the first cluster group and a second trending entity from the second cluster group.

3. The method of claim 1 , further comprising:

receiving a search query from a client application of a user device to search the messages posted to the messaging platform;

identifying a first trending entity from the first cluster group as matching a term of the search query; and

transmitting, in response to the search query, search results to the client application, the search results identifying the first trending entity and a second trending entity from the second cluster group.

4. The method of claim 1 , further comprising:

generating ranking information by ranking a plurality of cluster groups, including the first cluster group and the second cluster group, based on popularity metrics of trending entities associated with each cluster group; and

storing the ranking information in the memory device.

5. The method of claim 1 , further comprising:

retrieving the cluster chain from the memory device; and

linking the second cluster group with a third cluster group having at least two trending entities identified as similar to each other during a third period of time.

6. The method of claim 1 , further comprising:

assigning a cluster identifier to the first cluster group; and

assigning the cluster identifier of the first cluster group to the second cluster group in response to the second cluster group being linked to the first cluster group.

7. The method of claim 1 , wherein the one or more similarity-based clustering operations include:

generating a similarity graph based on similarity values associated with the plurality of trending entities, the similarity graph including nodes representing the plurality of trending entities and edges being annotated with the similarity values; and

partitioning the similarity graph according to a clustering algorithm to detect the first cluster group.

8. The method of claim 7 , wherein the one or more similarity-based clustering operations include:

computing the similarity values based on frequency count and co-occurrences among the plurality of trending entities over a time window; and

filtering the similarity graph based on a similarity threshold such that edges having similarity values less than the similarity threshold are removed from the similarity graph, wherein the filtered similarity graph is partitioned according to the clustering algorithm to detect the first cluster group.

9. The method of claim 8 , further comprising:

computing a first entity vector and a second entity vector for a pair of trending entities in the first cluster group, each of the first entity vector and the second entity vector including a series of values that indicate the frequency count and co-occurrences of a respective trending entity in the messages posted to the messaging platform; and

computing a similarity value between the first trending entity and the second trending entity by executing a cosine similarity function using the first entity vector and the second entity vector.

10. The method of claim 1 , wherein the second cluster group is determined as related to the first cluster group based on maximum weighted bipartite matching between the first cluster group and the second cluster group.

11. The method of claim 1 , further comprising:

computing an edge weight between the first cluster group and the second cluster group based on how many trending entities that are shared between the first cluster group and the second cluster group; and

generating the link between the first cluster group and the second cluster group in response to the edge weight being greater than a threshold level.

12. A messaging system for detecting real-time event, the messaging system comprising:

at least one processor; and

a non-transitory computer-readable medium storing executable instructions that when executed by the at least one processor cause the at least one processor to:

identify a plurality of trending entities from messages posted to a messaging platform, wherein a trending entity is an entity mentioned by users in the messages posted to the messaging platform at a rate higher than other entities;

execute one or more similarity-based clustering operations on the plurality of trending entities to detect a first cluster group of trending entities over a first period of time and a second cluster group of trending entities over a second period of time, the first cluster group including at least two trending entities identified as similar to each other during the first period of time, the second cluster group including at least two trending entities identified as similar to each other during the second period of time;

determine that the second cluster group is related to the first cluster group based on how many trending entities are shared between the first cluster group and the second cluster group;

generate a cluster chain by linking the second cluster group with the first cluster group, the cluster chain representing an event over the first and second periods of time; and

store the cluster chain in a memory device on the messaging platform.

13. The messaging system of claim 12 , wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to:

transmit digital data to a client application of a user device to render information about the event in a user interface of the client application, the information identifying a first trending entity from the first cluster group and a second trending entity from the second cluster group.

14. The messaging system of claim 12 , wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to:

generate ranking information by ranking a plurality of cluster groups, including the first cluster group and the second cluster group, based on an aggregate popularity metric associated with each cluster group; and

store the ranking information in the memory device.

15. The messaging system of claim 12 , wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to:

obtain a list of trending entities from a trend detector service;

extract entities from the messages posted to the messaging platform; and

filter the extracted entities using the list of trending entities to obtain the plurality of trending entities.

16. The messaging system of claim 12 , wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to:

assign a same cluster identifier to cluster groups of a single cluster chain.

17. The messaging system of claim 12 , wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to:

compute similarity values based on frequency count and co-occurrences among trending entities over a time window and generate a similarity graph based on the similarity values, the similarity graph including nodes representing the trending entities and edges being annotated with the similarity values.

18. The messaging system of claim 12 , wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to:

detect different terms that are used to describe the event on the messaging platform by periodically detecting and linking cluster groups over time.

19. The messaging platform of claim 12 , wherein the executable instructions include instructions that when executed by the at least one processor cause the at least one processor to:

retrieve the cluster chain from the memory device and link the second cluster group with a third cluster group having at least two trending entities identified as similar to each other during a third period of time.

20. A non-transitory computer-readable medium storing executable instructions that when executed by at least one processor are configured to cause the at least one processor to execute operations, the operations comprising:

identify a plurality of trending entities from messages posted to a messaging platform, wherein a trending entity is an entity mentioned by users in the messages posted to the messaging platform at a rate higher than other entities;

execute one or more similarity-based clustering operations on the plurality of trending entities to detect a first cluster group of trending entities over a first period of time and a second cluster group of trending entities over a second period of time, the first cluster group including at least two trending entities identified as similar to each other during the first period of time, the second cluster group including at least two trending entities identified as similar to each other during the second period of time;

determine that the second cluster group is related to the first cluster group;

generate a cluster chain by linking the second cluster group with the first cluster group, the cluster chain representing an event over the first and second periods of time;

store the cluster chain in a memory device of the messaging platform; and

transmit digital data to a client application of a user device to render information about the event in a user interface of the client application, the information identifying a first trending entity from the first cluster group and a second trending entity from the second cluster group.

21. The non-transitory computer-readable medium of claim 20 , wherein the operations include:

compute a popularity metric for each trending entity of a respective cluster group based on how many times a respective trending entity is mentioned in the messages posted to the messaging platform;

aggregate the popularity metrics for trending entities of the respective cluster group to compute an overall popularity metric for the respective cluster group;

generate ranking information by ranking a plurality of cluster groups, including the first cluster group and the second cluster group, based on the overall popularity metric associated with each cluster group; and

store the ranking information in the memory device.

22. The non-transitory computer-readable medium of claim 20 , wherein the operations include:

extract entities from the messages posted to the messaging platform, the entities including at least one of named entities or hashtags;

obtain a list of trending entities from a trend detector service via a server communication interface; and

identify the plurality of trending entities from the extracted entities based on the list of trending entities such that non-trending entities are filtered out of the extracted entities.

23. The non-transitory computer-readable medium of claim 20 , wherein the operations include:

assign a cluster identifier to the first cluster group; and

assign the cluster identifier of the first cluster group to the second cluster group in response to the second cluster group being linked to the first cluster group.

24. The non-transitory computer-readable medium of claim 20 , wherein the operations include:

compute similarity values based on frequency count and co-occurrences among the plurality of trending entities over a time window, each similarity value indicating a level of similarity between two trending entities;

generate a similarity graph based on the similarity values, the similarity graph including nodes representing the plurality of trending entities and edges being annotated with the similarity values;

filter the similarity graph based on a similarity threshold value such that edges having similarity values less than the similarity threshold are removed from the similarity graph; and

partition the filtered similarity graph according to a clustering algorithm to detect the first cluster group, the clustering algorithm including a Louvain algorithm.

25. The non-transitory computer-readable medium of claim 24 , wherein the operations include:

compute a first entity vector and a second entity vector for a pair of trending entities in the first cluster group, each of the first entity vector and the second entity vector including a series of values that indicate the frequency count and co-occurrences of a respective trending entity in the messages posted to the messaging platform; and

compute a similarity value between the first trending entity and the second trending entity by executing a cosine similarity function using the first entity vector and the second entity vector.

26. The non-transitory computer-readable medium of claim 20 , wherein the plurality of trending entities are identified by a burst detector, wherein the one or more similarity-based clustering operations are executed by a cluster chain detector, wherein the operations include:

independently monitor computing resources of the burst detector and computing resources of the cluster chain; and

adjust the computer resources of the cluster chain detector independently of the computer resources of the burst detector.

27. The non-transitory computer-readable medium of claim 26 , wherein one or more operations of the burst detector are performed in parallel with the one or more similarity-based clustering operations of the cluster chain detector.

28. The non-transitory computer-readable medium of claim 20 , wherein the operations include:

receive a search query from the client application to search the messages posted to the messaging platform;

identify the first trending entity from the first cluster group as matching a term of the search query; and

transmit, in response to the search query, search results to the client application, the search results identifying the first trending entity and the second trending entity from the second cluster group, the second trending entity not matching the term of the search query.