Methods and systems for anomaly and pattern detection of unstructured big data
A computing system includes: a memory, containing instructions for a method for anomaly and pattern detection of unstructured big data via semantic analysis and dynamic knowledge graph construction; a processor, coupled with the memory and, when the instructions being executed, configured to: receive unstructured big data associated with social network interactions, events, or activities; parse and structure the unstructured big data to generate structured big data; form a dynamic knowledge base based on the structured big data; and perform sematic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities; and a display, comprising an interactive graphical user interface (GUI), configured to receive the anomalies and patterns to display real-time actionable alerts, provide recommendations, and support decisions.
1 . A computing system, comprising:
a memory, containing instructions for a method for anomaly and pattern detection of unstructured big data via semantic analysis and dynamic knowledge graph construction; and
a processor, coupled with the memory and, when the instructions being executed, configured to:
receive unstructured big data associated with social network interactions, events, or activities;
parse and structure the unstructured big data to generate structured big data;
form a dynamic knowledge base based on the structured big data; and
perform semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities; and
a display, comprising an interactive graphical user interface (GUI), configured to receive the anomalies and patterns to present real-time actionable alerts, provide recommendations, and support decisions,
wherein performing semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities includes performing an Enhanced Heartbeat Graph (EHG)-based emerging event detection to predict an emerging event,
wherein performing the EHG-based emerging event detection includes:
generating word metrics series, including temporal aggregation of text stream and network generation of aggregated super-document,
generating EHG series, the EHG series being a set of graphs where each EHG is calculated from a pair of adjacent metrics in the word metrics series,
identifying strong events by performing a rule-based classification function based on extracting three features of events,
calculating ranking scores for words within a corresponding super-document of an EHG labeling strong to obtain a ranked list of keywords, and
determining an emerging event by finding a representative micro-document in a time period that the EHG labels strong, the representative micro-document being a micro-document with the highest relevance score in the time period, and a relevance score of a micro-document being calculated as a sum of a ranking score of each word in that corresponding micro-document, and
wherein performing semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities includes performing a social network centrality analysis process, and the social network centrality analysis process includes building a logistic function to include all of a degree centrality, a betweenness centrality, and a closeness centrality into the function when assigning a network score for an item in a social network.
2 . The system of claim 1 , wherein the unstructured big data comprises text, images, videos, Uniform Resource Locators (URLs), geolocations, timestamps, or contextual data.
3 . The system of claim 1 , wherein the unstructured big data comprises dynamic knowledge and static knowledge, the dynamic knowledge including open source streaming data and open source historical data, and the static knowledge including ground truth knowledge data.
4 . The system of claim 3 , wherein the processor is configured to store the static knowledge in a knowledge graph (KG) database (KGDB) and to store the dynamic knowledge into knowledge nuggets with a standard resource description framework (RDF) format.
5 . The system of claim 4 , wherein the processor is configured to fuse the knowledge nuggets and KGDB to form the dynamic knowledge base.
6 . The system of claim 1 , wherein the instructions comprise an automatic anomaly detection module for detecting the anomalies and a pattern discovery module for discovering the patterns.
7 . The system of claim 1 , wherein the dynamic knowledge base includes a text data-based knowledge graph or a social knowledge graph.
8 . The system of claim 1 , wherein the processor is configured to perform one or more of a social network centrality analysis process, or a behavior pattern analysis process.
9 . The method of claim 1 , wherein the network score for the item ν i is defined as:
p
(
v
i
)
=
1
1
+
exp
(
-
(
β
1
C
D
(
v
i
)
+
β
2
C
C
(
v
i
)
+
β
3
C
B
(
v
i
)
)
)
where β j , j∈[1,2,3] is a parameter to standardize values of the degree centrality, the betweenness centrality, and the closeness centrality.
10 . A computer-implemented method for anomaly and pattern detection of unstructured big data via semantic analysis and dynamic knowledge graph construction, performed by a hardware processor, comprising:
receiving unstructured big data associated with social network interactions, events, or activities;
parsing and structuring the unstructured big data to generate structured big data;
forming a dynamic knowledge base based on the structured big data;
performing semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities; and
feeding the anomalies and patterns into an interactive graphical user interface (GUI), to display real-time actionable alerts, provide recommendations, and support decisions,
wherein performing semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities includes performing an Enhanced Heartbeat Graph (EHG)-based emerging event detection to predict an emerging event,
wherein performing the EHG-based emerging event detection includes:
generating word metrics series, including temporal aggregation of text stream and network generation of aggregated super-document,
generating EHG series, the EHG series being a set of graphs where each EHG is calculated from a pair of adjacent metrics in the word metrics series,
identifying strong events by performing a rule-based classification function based on extracting three features of events,
calculating ranking scores for words within a corresponding super-document of an EHG labeling strong to obtain a ranked list of keywords, and
determining an emerging event by finding a representative micro-document in a time period that the EHG labels strong, the representative micro-document being a micro-document with the highest relevance score in the time period, and a relevance score of a micro-document being calculated as a sum of a ranking score of each word in that corresponding micro-document, and
wherein performing semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities includes performing a social network centrality analysis process, and the social network centrality analysis process includes building a logistic function to include all of a degree centrality, a betweenness centrality, and a closeness centrality into the function when assigning a network score for an item in a social network.
11 . The method of claim 10 , wherein forming a dynamic knowledge base based on the structured big data, comprises:
performing triple extraction from text data of the structured big data;
constructing a text data-based knowledge graph (KG); and
constructing a social knowledge graph (SKG).
12 . The method of claim 11 , wherein the triple extraction includes name entity recognition (NER) and predicate recognition.
13 . The method of claim 10 , wherein performing semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities, comprises:
performing a behavior pattern analysis process,
wherein the processes are configured to work in conjunction to enhance anomaly and pattern detection.
14 . The method of claim 13 , wherein the automatic fact-checking process includes information retrieval (IR), natural language process (NLP) techniques, or network/graph theory.
15 . The method of claim 13 , wherein the automatic fact-checking process comprises: knowledge inference.
16 . The method of claim 15 , wherein the knowledge inference includes a semantic proximity method.
17 . The method of claim 13 , wherein the emerging event detection process includes a feature pivot graph-based event detection method.
18 . The method of claim 13 , wherein the behavior pattern analysis process comprises an association rule method.
19 . The method of claim 10 , wherein the unstructured big data comprises text, images, videos, Uniform Resource Locators (URLs), geolocations, timestamps, or contextual data.
20 . A non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for anomaly and pattern detection of unstructured big data via semantic analysis and dynamic knowledge graph construction, the method comprising:
receiving unstructured big data associated with social network interactions, events, or activities;
parsing and structuring the unstructured big data to generate structured big data;
forming a dynamic knowledge base based on the structured big data;
performing semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities; and
feeding the anomalies and patterns into an interactive graphical user interface (GUI), to display real-time actionable alerts, provide recommendations, and support decisions,
wherein performing semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities includes performing an Enhanced Heartbeat Graph (EHG)-based emerging event detection to predict an emerging event,
wherein performing the EHG-based emerging event detection includes:
generating word metrics series, including temporal aggregation of text stream and network generation of aggregated super-document,
generating EHG series, the EHG series being a set of graphs where each EHG is calculated from a pair of adjacent metrics in the word metrics series,
identifying strong events by performing a rule-based classification function based on extracting three features of events,
calculating ranking scores for words within a corresponding super-document of an EHG labeling strong to obtain a ranked list of keywords, and
determining an emerging event by finding a representative micro-document in a time period that the EHG labels strong, the representative micro-document being a micro-document with the highest relevance score in the time period, and a relevance score of a micro-document being calculated as a sum of a ranking score of each word in that corresponding micro-document, and
wherein performing semantic reasoning on the dynamic knowledge base to discover patterns and anomalies among the social network interactions, events, or activities includes performing a social network centrality analysis process, and the social network centrality analysis process includes building a logistic function to include all of a degree centrality, a betweenness centrality, and a closeness centrality into the function when assigning a network score for an item in a social network.