Log anomaly detection using temporal-attentive dynamic graphs
A computer-implemented method for employing a graph-based log anomaly detection framework to detect relational anomalies in system logs is provided. The method includes collecting log events from systems or applications or sensors or instruments, constructing dynamic graphs to describe relationships among the log events and log fields by using a sliding window with a fixed time interval to snapshot a batch of the log events, capturing sequential patterns by employing temporal-attentive transformers to learn temporal dependencies within the sequential patterns, and detecting anomalous patterns in the log events based on relationships between the log events and temporal context determined from the temporal-attentive transformers.
1 . A computer-implemented method for employing a graph-based log anomaly detection framework to detect relational anomalies in system logs, the method comprising:
collecting log events from systems or applications or sensors or instruments;
constructing dynamic graphs to describe relationships among the log events and log fields by using a sliding window with a fixed time interval to snapshot a batch of the log events, wherein a log anomaly detection framework detects relational anomalies in system logs and incorporates log semantics, relationship patterns, and sequential patterns into a unified framework for anomaly detection;
capturing sequential patterns by employing temporal-attentive transformers to learn temporal dependencies within the sequential patterns, wherein each subsequence includes unique local information for determining whether a whole sequence is abnormal or not; and
detecting anomalous patterns in the log events based on relationships between the log events and temporal context determined from the temporal-attentive transformers and representations in the dynamic graphs whose distance to a center ranks at a specific percentile beyond a decision boundary radius including a soft-margin decision boundary determined by an ad-hoc heuristic.
2 . The computer-implemented method of claim 1 , wherein a field extractor employs prompt-based few-shot learning to extract the log fields.
3 . The computer-implemented method of claim 1 , wherein the dynamic graphs are encoded in a latent space by using graph convolutional network (GCN) encoders followed by a pre-trained transformer.
4 . The computer-implemented method of claim 3 , wherein the GCN encoders distinguish between normal and anomalous edges with structural, semantic, and sequential information.
5 . The computer-implemented method of claim 3 , wherein encoded hidden states for each of the log events and log fields is regarded as an attribute and an adjacency matrix represents a structure of the dynamic graphs.
6 . The computer-implemented method of claim 1 , wherein, for each dynamic graph, negative edges are sampled and edge scores are calculated based on learned hidden states.
7 . The computer-implemented method of claim 6 , wherein a pair-wise margin loss is utilized to minimize positive edge scores and to maximize negative edge scores following a one-class training objective.
8 . A computer program product for employing a graph-based log anomaly detection framework to detect relational anomalies in system logs, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
collecting log events from systems or applications or sensors or instruments;
constructing dynamic graphs to describe relationships among the log events and log fields by using a sliding window with a fixed time interval to snapshot a batch of the log events, wherein a log anomaly detection framework detects relational anomalies in system logs and incorporates log semantics, relationship patterns, and sequential patterns into a unified framework for anomaly detection;
capturing sequential patterns by employing temporal-attentive transformers to learn temporal dependencies within the sequential patterns, wherein each subsequence includes unique local information for determining whether a whole sequence is abnormal or not; and
detecting anomalous patterns in the log events based on relationships between the log events and temporal context determined from the temporal-attentive transformers and representations in the dynamic graphs whose distance to a center ranks at a specific percentile beyond a decision boundary radius including a soft-margin decision boundary determined by an ad-hoc heuristic.
9 . The computer program product of claim 8 , wherein a field extractor employs prompt-based few-shot learning to extract the log fields.
10 . The computer program product of claim 8 , wherein the dynamic graphs are encoded in a latent space by using graph convolutional network (GCN) encoders followed by a pre-trained transformer.
11 . The computer program product of claim 10 , wherein the GCN encoders distinguish between normal and anomalous edges with structural, semantic, and sequential information.
12 . The computer program product of claim 10 , wherein encoded hidden states for each of the log events and log fields is regarded as an attribute and an adjacency matrix represents a structure of the dynamic graphs.
13 . The computer program product of claim 8 , wherein, for each dynamic graph, negative edges are sampled and edge scores are calculated based on learned hidden states.
14 . The computer program product of claim 13 , wherein a pair-wise margin loss is utilized to minimize positive edge scores and to maximize negative edge scores following a one-class training objective.
15 . A computer processing system for employing a graph-based log anomaly detection framework to detect relational anomalies in system logs, comprising:
a memory device for storing program code; and
a processor device, operatively coupled to the memory device, for running the program code to:
collect log events from systems or applications or sensors or instruments;
construct dynamic graphs to describe relationships among the log events and log fields by using a sliding window with a fixed time interval to snapshot a batch of the log events, wherein a log anomaly detection framework detects relational anomalies in system logs and incorporates log semantics, relationship patterns, and sequential patterns into a unified framework for anomaly detection;
capture sequential patterns by employing temporal-attentive transformers to learn temporal dependencies within the sequential patterns, wherein each subsequence includes unique local information for determining whether a whole sequence is abnormal or not; and
detect anomalous patterns in the log events based on relationships between the log events and temporal context determined from the temporal-attentive transformers and representations in the dynamic graphs whose distance to a center ranks at a specific percentile beyond a decision boundary radius including a soft-margin decision boundary determined by an ad-hoc heuristic.
16 . The computer processing system of claim 15 , wherein a field extractor employs prompt-based few-shot learning to extract the log fields.
17 . The computer processing system of claim 15 , wherein the dynamic graphs are encoded in a latent space by using graph convolutional network (GCN) encoders followed by a pre-trained transformer.
18 . The computer processing system of claim 17 , wherein the GCN encoders distinguish between normal and anomalous edges with structural, semantic, and sequential information.
19 . The computer processing system of claim 17 , wherein encoded hidden states for each of the log events and log fields is regarded as an attribute and an adjacency matrix represents a structure of the dynamic graphs.
20 . The computer processing system of claim 15 , wherein, for each dynamic graph, negative edges are sampled and edge scores are calculated based on learned hidden states.