Measure-based anomaly detection
Method and system for detecting an unknown undesirable event, such as (but not limited to) a cyber-threat, a cyber-intrusion, a financial fraud event or a monitored process malfunction of breakdown. An exemplary method embodiment comprises obtaining a dataset comprising a plurality n of multidimensional data points with a dimension m≥2 wherein each data point is a vector of m features, processing the MDPs using measure-based diffusion maps to embed the MDPs into a lower dimension embedded space, and detecting in the embedded space an abnormal MDP without relying on a signature of a threat, the abnormal MDP being indicative of the unknown undesirable event.
1. A computer system, comprising:
a) a processing module configurable to receive data comprising a plurality of measurements, each measurement being a N-dimensional data point (MDP) wherein N≥3, the processing module further configurable to utilize a measure-based Gaussian correlation (MGC) kernel
K
(
x
,
y
)
=
∑
r
∈
M
exp
(
-
x
-
r
/
ɛ
)
exp
(
-
y
-
r
/
ɛ
)
q
(
r
)
,
x, y∈X to generate an associated distribution;
b) a detection module configurable to detect an abnormal measurement using a plurality of representatives chosen from the associated distribution, wherein the plurality of chosen representatives is smaller than the plurality of measurements; and
c) a configuration and operation server operative to configure the processing and detection modules,
whereby the use of the MGC kernel and of only chosen representatives instead of the plurality of measurements enables the computer system to detect abnormal measurements faster and more efficiently.
2. The computer system of claim 1 , wherein the MGC kernel includes a measure assumption used to capture non-manifold, locally low-dimensional structures with varying local dimensionality in a space embedding the MDPs that has a lower dimension than N, and wherein the detection module is configurable to detect the abnormal measurement without finding a manifold geometry.
3. The computer system of claim 2 , wherein the detection module is configurable to detect the abnormal measurement independently of an intrinsic geometry of the data.
4. The computer system of claim 2 , wherein the data includes financial data and wherein the abnormal measurement is indicative of financial fraud.
5. The computer system of claim 2 , wherein the data includes financial data and wherein the abnormal measurement is indicative of money laundering.
6. The computer system of claim 1 , wherein the detection module is configurable to detect the abnormal measurement independently of an intrinsic geometry of the data.
7. The computer system of claim 1 , wherein the detection module is further configured to detect the abnormal measurement automatically and without supervision.
8. The computer system of claim 1 , wherein the detection module is further configured to detect the abnormal measurement without use of a rule.
9. The computer system of claim 1 , wherein the detection module is further configured to detect the abnormal measurement without use of a domain expert.
10. The computer system of claim 1 , wherein the data includes financial data and wherein the abnormal measurement is indicative of financial fraud.
11. The computer system of claim 1 , wherein the data includes financial data and wherein the abnormal measurement is indicative of money laundering.
12. A method, comprising:
a) providing a computer system comprising a processing module, a detection module and a configuration and detection server; and
b) by the configuration and detection server:
i. configuring the processing module to receive data comprising a plurality of measurements, each measurement being a N-dimensional data point (MDP) wherein N≥3, and to utilize a measure-based Gaussian correlation (MGC) kernel
K
(
x
,
y
)
=
∑
r
∈
M
exp
(
-
x
-
r
/
ɛ
)
exp
(
-
y
-
r
/
ɛ
)
q
(
r
)
,
x, y∈X to generate an associated distribution, and
ii. configuring the detection module detect an abnormal measurement using a plurality of representatives chosen from the associated distribution, wherein the plurality of chosen representatives is smaller than the plurality of measurements,
whereby the use of the MGC kernel and of only chosen representatives instead of the plurality of measurements enables the computer system to detect abnormal measurements faster and more efficiently.
13. The method of claim 12 , wherein the MGC kernel includes a measure assumption used to capture non-manifold, locally low-dimensional structures with varying local dimensionality in a space embedding the MDPs that has a lower dimension than N, and wherein the configuring the detection module to detect an abnormal measurement includes configuring the detection module to detect the abnormal measurement without finding a manifold geometry.
14. The method of claim 13 , wherein the configuring the detection module to detect an abnormal measurement includes configuring the detection module to detect the abnormal measurement independently of an intrinsic geometry of the data.
15. The method of claim 12 , wherein the configuring the detection module to detect an abnormal measurement includes configuring the detection module to detect the abnormal measurement independently of an intrinsic geometry of the data.
16. The method of claim 12 , wherein the configuring the detection module to detect an abnormal measurement includes configuring the detection module to detect the abnormal measurement automatically and without supervision.
17. The method of claim 12 , wherein the configuring the detection module to detect an abnormal measurement includes configuring the detection module to detect the abnormal measurement without use of a rule.
18. The method of claim 12 , wherein the configuring the detection module to detect an abnormal measurement includes configuring the detection module to detect the abnormal measurement without use of a domain expert.
19. The method of claim 12 , wherein the data includes financial data and wherein the abnormal measurement is indicative of financial fraud.
20. The method of claim 12 , wherein the data includes financial data and wherein the abnormal measurement is indicative of money laundering.