Method and apparatus for anomaly detection
An apparatus for anomaly detection, the apparatus comprising means for: Collecting a plurality of log messages from a data processing system, log messages comprising textual content and numeric attributes, Classifying the plurality of log messages into a plurality of clusters as a function of a number of the numeric attributes in the log messages, such that the log messages within a cluster have a given number of the numeric attributes, For at least one of the clusters, computing at least one encoding vector associated to a numeric attribute, Computing a combined semantic embedding vector from the textual contents of the plurality of log messages, Combining the at least one encoding vector with the combined semantic embedding vector into a final encoding vector, and Feeding the final encoding vector to an anomaly detection module intended to detect an anomaly in the data processing system.
1 . An apparatus to detect anomalies, the apparatus comprising:
an anomaly detection module;
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform,
collecting a plurality of log messages from a data processing system, the plurality of log messages reporting events that occur as the data processing system operates, the plurality of log messages comprising textual content and numeric attributes;
classifying the plurality of log messages into a plurality of clusters as a first function of a number of the numeric attributes in the plurality of log messages, such that the plurality of log messages within a cluster have a given number of the numeric attributes;
for at least one of the clusters, computing at least one encoding vector associated to a numeric attribute, wherein the at least one encoding vector is computed as a second function of a representative value of the numeric attribute across the plurality of log messages within the cluster and of a rank of the numeric attribute, wherein the rank is representative of an order of the numeric attribute within the given number of the numeric attributes of the plurality of log messages within the cluster;
computing a combined semantic embedding vector from textual contents of the plurality of log messages;
combining the at least one encoding vector with the combined semantic embedding vector into a final encoding vector; and
feeding the final encoding vector to the anomaly detection module configured to detect an anomaly in the data processing system.
2 . The apparatus according to claim 1 , wherein the apparatus is further caused to perform:
computing the at least one encoding vector associated to the numeric attribute for the plurality of clusters.
3 . The apparatus according to claim 1 , wherein the apparatus is further caused to perform:
computing a plurality of encoding vectors associated to a plurality of the numeric attributes for the at least one of the clusters.
4 . The apparatus according to claim 3 , wherein the apparatus is further caused to perform:
combining the plurality of encoding vectors with the combined semantic embedding vector into the final encoding vector.
5 . The apparatus according to claim 1 , wherein the apparatus is further caused to perform:
computing at least one additional numeric attribute from the plurality of log messages of the at least one of the clusters;
appending the at least one additional numeric attribute to the numeric attributes within the at least one of the clusters; and
increasing the given number of the numeric attributes of the plurality of log messages within the cluster.
6 . The apparatus according to claim 1 , wherein the apparatus is further caused to compute the plurality of clusters as a third function of the textual contents in the plurality of log messages.
7 . The apparatus according to claim 1 , wherein the apparatus is further caused to perform:
computing a plurality of semantic embedding vectors from the textual contents; and
combining the plurality of semantic embedding vectors into the combined semantic embedding vector.
8 . The apparatus according to claim 1 , wherein the apparatus is further caused to convert a plurality of average values into high dimensional polar coordinates.
9 . The apparatus according to claim 1 , wherein the apparatus is further caused to,
compute an intermediate encoding vector from the representative value of the numeric attribute;
select a projection vector within a basis of orthogonal vectors as a third function of the rank of the numeric attribute; and
compute the at least one encoding vector by projecting the intermediate encoding vector on the projection vector.
10 . The apparatus according to claim 1 , wherein the apparatus is further caused to,
compute a transitional encoding vector from the representative value of the numeric attribute; and
multiply at least one coordinate of the transitional encoding vector by a factor depending on an index of the at least one coordinate and the rank of the numeric attribute.
11 . The apparatus according to claim 1 , wherein the apparatus is further caused to perform:
returning an anomaly detection signal to the data processing system in order to cause a corrective action in the data processing system, the anomaly detection signal being computed by the anomaly detection module.
12 . The apparatus according to claim 1 , wherein the anomaly detection module is an autoencoder.
13 . The apparatus according to claim 12 , wherein the apparatus is further caused to perform:
receiving a reconstruction loss computed by the autoencoder; and
classifying a value of the reconstruction loss higher than an anomaly threshold as an anomaly.
14 . The apparatus according to claim 1 , wherein the apparatus is further caused to perform:
training the anomaly detection module using a plurality of training encoding vectors, the plurality of training encoding vectors being computed from a training dataset of log messages.
15 . A method of operating an apparatus for anomaly detection, the method comprising:
collecting a plurality of log messages from a data processing system, the plurality of log messages reporting events that occur as the data processing system operates, the plurality of log messages comprising textual content and numeric attributes;
classifying the plurality of log messages into a plurality of clusters as a first function of a number of the numeric attributes in the plurality of log messages, such that the plurality of log messages within a cluster have a given number of the numeric attributes;
for at least one of the clusters, computing at least one encoding vector associated to a numeric attribute, wherein the at least one encoding vector is computed as a second function of a representative value of the numeric attribute across the plurality of log messages within the cluster and of a rank of the numeric attribute, wherein the rank is representative of an order of the numeric attribute within the given number of the numeric attributes of the plurality of log messages within the cluster;
computing a combined semantic embedding vector from textual contents of the plurality of log messages;
combining the at least one encoding vector with the combined semantic embedding vector into a final encoding vector; and
feeding the final encoding vector to an anomaly detection module of the apparatus, the anomaly detection module configured to detect an anomaly in the data processing system.
16 . The method according to claim 15 , further comprising returning an anomaly detection signal representing the anomaly.
17 . The method according to claim 16 , further comprising performing corrective action as a third function of the anomaly detection signal.
18 . The method according to claim 17 , wherein the corrective action is performed in the data processing system.