System and method for anomaly detection in dynamically evolving data using low rank matrix decomposition
Detection of abnormalities in HDBD is performed by processing it to obtain a dictionary from a training data. This is done by computing a low rank randomized LU decomposition which enables constant online updating of the training data and thus gets constant updating of the normal profile in the background.
1. In a computer system, a method comprising:
a) receiving training data in the form of a m×n matrix A with rank k, wherein matrix A comprises a plurality of m multidimensional data points (MDDPs) with a dimension n≥3;
b) based on the training data, constructing a dictionary D by applying randomized lower and upper triangular matrix (LU) decomposition to matrix A, wherein dictionary D is of size m×k or n×k and wherein k<n and k<<m;
c) determining a score S for a newly arrived multidimensional data point (NAMDDP) x based on the constructed dictionary D and its pseudo-inverse D † using data of size m×k or n×k instead of m×n, wherein score S is in the form
S
←
DD
†
x
-
x
β
(
x
)
and wherein β(x) is a normalization function; and
d) classifying x as a normal MDDP or as an anomaly by comparing its score S with a threshold T, wherein classification of x as an anomaly is indicative of detection of an unknown undesirable event,
wherein the determining a score S by processing dictionary D of size m×k or n×k instead of matrix A of size m×n causes the determining of score S to be faster and to require less computational effort, thereby enhancing anomaly detection performance, and
wherein the constructing a dictionary D, determining a score S and classifying x as a normal MDDP or as an anomaly is done by an anomaly detection engine of the computer system that is configurable on-the-fly without changing software code and without even halting operation of the computer system to optimize work flow, throughput and performance of the computer system for anomaly detection.
2. The method of claim 1 , wherein the threshold T is determined based on a percentage of NAMDPPs with the largest score S.
3. The method of claim 1 , wherein the threshold T is determined based on of NAMDPPs that have a score S above a certain limit.
4. The method of claim 1 , wherein the threshold T is determined by comparing between scores S 1 and S 2 of two consecutive NAMDDPs.
5. The method of claim 1 , wherein the unknown undesirable event is selected form the group consisting of a financial risk event, a financial threat event, a financial fraud event and a financial network intrusion event.
6. The method of claim 1 , wherein the unknown undesirable event includes money laundering.
7. The method of claim 1 , wherein the unknown undesirable event is an undesirable event that is unknown before application of the method to perform steps (a)-d) to detect and classify the anomaly indicative of the detection of the unknown undesirable event.
8. The method of claim 1 , wherein the classifying is performed automatically and unsupervised without relying on a signature.
9. The method of claim 1 , wherein the classifying is performed automatically and unsupervised without relying on a rule.
10. The method of claim 1 , wherein the classifying is performed automatically and unsupervised without relying on domain expertise.
11. A non-transitory computer readable storage medium storing a set of instructions that are executable by at least one processor of a server in a computer system to perform a method for detecting an anomaly, the method comprising:
a) receiving training data in the form of a m×n matrix A with rank k, wherein matrix A comprises a plurality of m multidimensional data points (MDDPs) with a dimension n≥3;
b) based on the training data, constructing a dictionary D by applying randomized lower and upper triangular matrix (LU) decomposition to matrix A, wherein dictionary D is of size m×k or n×k and wherein k<n and k<<m;
c) determining a score S for a newly arrived multidimensional data point (NAMDDP) x based on the constructed dictionary D and its pseudo-inverse D † using data of size m×k or n×k instead of m×n, wherein score S is in the form
S
←
DD
†
x
-
x
β
(
x
)
and wherein β(x) is a normalization function; and
d) classifying x as a normal MDDP or as an anomaly by comparing its score S with a threshold T, wherein classification of x as an anomaly is indicative of detection of an unknown undesirable event,
wherein the determining a score S by processing dictionary D of size m×k or n×k instead of matrix A of size m×n causes the determining of score S to be faster and to require less computational effort, thereby enhancing anomaly detection performance, and
wherein the constructing a dictionary D, determining a score S and classifying x as a normal MDDP or as an anomaly is done by an anomaly detection engine of the computer system that is configurable on-the-fly without changing software code and without even halting operation of the computer system to optimize work flow, throughput and performance of the computer system for anomaly detection.
12. The non-transitory computer readable storage medium of claim 11 , wherein the threshold T is determined based on a percentage of NAMDPPs with the largest score S.
13. The non-transitory computer readable storage medium of claim 11 , wherein the threshold T is determined based on of NAMDPPs that have a score S above a certain limit.
14. The non-transitory computer readable storage medium of claim 11 , wherein the threshold T is determined by comparing between scores S 1 and S 2 of two consecutive NAMDDPs.
15. The non-transitory computer readable storage medium of claim 11 , wherein the unknown undesirable event is selected form the group consisting of a financial risk event, a financial threat event, a financial fraud event and a financial network intrusion event.
16. The non-transitory computer readable storage medium of claim 11 , wherein the unknown undesirable event includes money laundering.
17. The non-transitory computer readable storage medium of claim 11 , wherein the unknown undesirable event is an undesirable event that is unknown before application of the method to perform steps (a)-d) to detect and classify the anomaly indicative of the detection of the unknown undesirable event.
18. The non-transitory computer readable storage medium of claim 11 , wherein the classifying is performed automatically and unsupervised without relying on a signature.
19. The non-transitory computer readable storage medium of claim 11 , wherein the classifying is performed automatically and unsupervised without relying on a rule.
20. The non-transitory computer readable storage medium of claim 11 , wherein the classifying is performed automatically and unsupervised without relying on domain expertise.