Anomaly detection using a semi-supervised locally adaptive similarity kernel
A method of detecting anomalies in data, includes receiving a dataset with a plurality of multidimensional data points (MDDPs) wherein a portion of the plurality of MDDPs are labeled and wherein other MDDPs of the plurality of MDDPs are unlabeled; based on a neighborhood size k of the plurality of MDDPs, computing a neighborhood radius ox for each MDDP in a reference dataset computed for the plurality of MDDPs; and generating a locally adaptive similarity (LAS) kernel of a newly arrived MDDP (NAMDDP) based on the neighborhood radius ox. The method additionally includes applying a random walk model to the LAS kernel to determine a probability of the NAMDDP being an anomaly; and if the NAMDDP is an anomaly, outputting data associated with an alarm or notification responsive to the detection of the anomaly.
1 . A method of detecting anomalies in data, comprising:
receiving a dataset comprising a plurality of multidimensional data points (MDDPs) wherein a portion of the plurality of MDDPs are labeled and wherein other MDDPs of the plurality of MDDPs are unlabeled;
based on a neighborhood size k of the plurality of MDDPs, computing a neighborhood radius σ x for each MDDP in a reference dataset computed for the plurality of MDDPs;
generating a locally adaptive similarity (LAS) kernel of a newly arrived MDDP (NAMDDP) based on the neighborhood radius σx;
applying a single step random walk model to the LAS kernel to determine a probability of the NAMDDP being an anomaly; and
if the NAMDDP is an anomaly, outputting data associated with an alarm or notification responsive to the detection of the anomaly.
2 . The method of claim 1 , wherein the computing a neighborhood radius σ x for each MDDP in a reference dataset computed for the plurality of MDDPs based on a neighborhood size k of the plurality of MDDPs, includes:
receiving the neighborhood size k for the plurality of MDDPs;
computing the reference dataset for the plurality of MDDPs; and
computing the neighborhood radius σ, for each MDDP in the reference dataset based on the neighborhood size k.
3 . The method of claim 1 , further comprising applying a k-nearest neighbors (KNN) algorithm to each MDDP in the reference dataset for use in the generating of the LAS kernel.
4 . The method of claim 1 , further comprising assigning to each MDDP and/or NAMDDP a score that reflects a normality in a probability assignment between 0 and 1 based on a scoring function S.
5 . The method of claim 4 , wherein a score equal to or approximating 0 is associated with a normal data point, and a score equal to or approximating 1 is associated with an abnormal data point.
6 . The method of claim 1 , further comprising assigning values of −1, 0, and 1 to the data points in the reference data set, wherein the value −1 is associated with a normal data point, the value 1 is associated with an abnormal data point, and the value 0 is associated with an unknown data point.
7 . The method of claim 1 , wherein the LAS kernel is described by a similarity measure S: n × n →[0,1].
8 . A computer program product, comprising: a non-transitory tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method of detecting anomalies in data that includes:
receiving a dataset comprising a plurality of multidimensional data points (MDDPs) wherein a portion of the plurality of MDDPs are labeled and wherein other MDDPs of the plurality of MDDPs are unlabeled;
based on a neighborhood size k of the plurality of MDDPs, computing a neighborhood radius σ x for each MDDP in a reference dataset computed for the plurality of MDDPs;
generating a locally adaptive similarity (LAS) kernel of a newly arrived MDDP (NAMDDP) based on the neighborhood radius σ x ;
applying a single step random walk model to the LAS kernel to determine a probability of the NAMDDP being an anomaly; and
if the NAMDDP is an anomaly, outputting data associated with an alarm or notification responsive to the detection of the anomaly.
9 . The computer program product of claim 8 , wherein the computing a neighborhood radius σ x for each MDDP in a reference dataset computed for the plurality of MDDPs based on a neighborhood size k of the plurality of MDDPs, includes:
receiving the neighborhood size k for the plurality of MDDPs;
computing the reference dataset for the plurality of MDDPs; and
computing the neighborhood radius σ x for each MDDP in the reference dataset based on the neighborhood size k.
10 . The computer program product of claim 8 , wherein the method further includes applying a k-nearest neighbors (KNN) algorithm to each MDDP in the reference dataset for use in the generating of the LAS kernel.
11 . The computer program product of claim 8 wherein the method further includes assigning to each MDDP and/or NAMDDP a score that reflects a normality in a probability assignment between 0 and 1 based on a scoring function S.
12 . The computer program product of claim 11 , wherein a score equal to or approximating 0 is associated with a normal data point, and a score equal to or approximating 1 is associated with an abnormal data point.
13 . The computer program product of claim 8 , wherein the method further includes assigning values of −1, 0, and 1 to the data points in the reference data set, wherein the value −1 is associated with a normal data point, the value 1 is associated with an abnormal data point, and the value 0 is associated with an unknown data point.
14 . The computer program product of claim 8 , wherein the LAS kernel is described by a similarity measure S: n × n ≥[0,1].
15 . A computer system, comprising: a hardware processor configurable to perform a method for detecting anomalies in data that includes:
receiving a dataset comprising a plurality of multidimensional data points (MDDPs) wherein a portion of the plurality of MDDPs are labeled and wherein other MDDPs of the plurality of MDDPs are unlabeled;
based on a neighborhood size k of the plurality of MDDPs, computing a neighborhood radius σ x for each MDDP in a reference dataset computed for the plurality of MDDPs;
generating a locally adaptive similarity (LAS) kernel of a newly arrived MDDP (NAMDDP) based on the neighborhood radius σ x ;
applying a single step random walk model to the LAS kernel to determine a probability of the NAMDDP being an anomaly; and
if the NAMDDP is an anomaly, outputting data associated with an alarm or notification responsive to the detection of the anomaly.
16 . The computer system of claim 15 , wherein the computing a neighborhood radius σx for each MDDP in a reference dataset computed for the plurality of MDDPs based on a neighborhood size k of the plurality of MDDPs, includes:
receiving the neighborhood size k for the plurality of MDDPs;
computing the reference dataset for the plurality of MDDPs; and
computing the neighborhood radius σ x for each MDDP in the reference dataset based on the neighborhood size k.
17 . The computer system of claim 15 , wherein the method further includes applying a k-nearest neighbors (KNN) algorithm to each MDDP in the reference dataset for use in the generating of the LAS kernel.
18 . The computer system of claim 15 , wherein the method further includes assigning to each MDDP and/or NAMDDP a score that reflects a normality in a probability assignment between 0 and 1 based on a scoring function S.
19 . The computer system of claim 15 , wherein the method further includes assigning values of −1, 0, and 1 to the data points in the reference data set, wherein the value −1 is associated with a normal data point, the value 1 is associated with an abnormal data point, and the value 0 is associated with an unknown data point.
20 . The computer system of claim 15 , wherein the LAS kernel is described by a similarity measure S: n × n →[0,1].