System and method for anomaly detection in dynamically evolving data using hybrid decomposition

1. A method, comprising steps of: in a computer system

a) receiving data comprising a plurality of measurements m of multidimensional data points (MDDPs), each data point having n features wherein n>3, wherein m>>n and wherein the data forms a matrix A=m×n;

b) applying randomized lower and upper triangular matrix (LU) decomposition to matrix A to obtain a dictionary D in the form of a matrix k×n where k is the rank of matrix A and k<<m;

c) embedding dictionary D in a lower dimension space that has a dimension lower than n, thereby obtaining an embedded dictionary D;

d) in the lower dimension space and based on embedded dictionary D, classifying a MDDP or a newly arrived MDDP (NAMDDP) as an anomaly;

e) preparing an alert related to the anomaly; and

f) sending the alert to a data client system,

whereby the combined reduction in the size of both m and n reduces computer processing requirements, thereby enhancing anomaly detection performance.

2. The method of claim 1 , further comprising:

g) repeating step (b) p≥1 times to obtain p additional dictionaries D(p);

h) concatenating each additional dictionary D(p) with all previously obtained dictionaries into a concatenated dictionary Dc; and

i) embedding concatenated dictionary Dc into the lower dimension space to obtain an embedded concatenated dictionary Dc, wherein the classifying a MDDP or NAMDDP as an anomaly includes classifying the MDDP or NAMDDP as an anomaly based on embedded concatenated dictionary Dc,

whereby the repeating step (b) p≥1 times removes randomization effects related to the randomized LU decomposition by causing eigenvalues of each dictionary D to decline, thereby providing a more robust embedded concatenated dictionary Dc and improving anomaly detection performance.

3. The method of claim 2 , wherein the applying randomized LU decomposition to obtain a dictionary D includes applying randomized LU to matrix A to obtain two matrices L and P and constructing dictionary D from the multiplication of matrix L with a transpose of matrix P.

4. The method of claim 2 , wherein the applying of randomized LU decomposition to obtain a dictionary D and the embedding of dictionary D in a lower dimension embedded space include applying the randomized LU decomposition and embedding dictionary D by dynamically configuring computer system parameters on-the-fly, whereby the dynamic on-the-fly configuring of computer system parameters is done without changing computer system software code and without halting computer system operation, thereby significantly improving computer system performance.

5. The method of claim 2 , wherein the classifying a MDDP or a newly arrived MDDP (NAMDDP) as an anomaly includes classifying without relying on a signature and/or a rule and/or on domain expertise.

6. The method of claim 1 , wherein the applying randomized LU decomposition to obtain a dictionary D includes applying randomized LU to matrix A to obtain two matrices L and P and constructing dictionary D from the multiplication of matrix L with a transpose of matrix P.

7. The method of claim 1 , wherein the applying of randomized LU decomposition to obtain a dictionary D and the embedding of dictionary D in a lower dimension embedded space include applying the randomized LU decomposition and embedding dictionary D by dynamically configuring computer system parameters on-the-fly, whereby the dynamic on-the-fly configuring of computer system parameters is done without changing computer system software code and without halting computer system operation, thereby significantly improving computer system performance.

8. The method of claim 1 , wherein the classifying a MDDP or a newly arrived MDDP (NAMDDP) as an anomaly includes classifying without relying on a signature and/or a rule and/or on domain expertise.

9. A computer program product, comprising: a non-transitory tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising:

a) receiving data comprising a plurality of measurements m of multidimensional data points (MDDPs), each data point having n features wherein n>3, wherein m>>n and wherein the data forms a matrix A=m×n;

b) applying randomized lower and upper triangular matrix (LU) decomposition to matrix A to obtain a dictionary D in the form of a matrix k×n where k is the rank of matrix A and k<<m;

c) embedding dictionary D in a lower dimension space that has a dimension lower than n, thereby obtaining an embedded dictionary D;

d) in the lower dimension space and based on embedded dictionary D, classifying a MDDP or a newly arrived MDDP (NAMDDP) as an anomaly;

e) preparing an alert related to the anomaly; and

f) sending the alert to a data client system,

whereby the combined reduction in the size of both m and n reduces computer processing requirements, thereby enhancing anomaly detection performance.

10. The computer program product of claim 9 , wherein the method further comprises:

g) repeating step (b) p≥1 times to obtain p additional dictionaries D(p);

h) concatenating each additional dictionary D(p) with all previously obtained dictionaries into a concatenated dictionary Dc; and

i) embedding concatenated dictionary Dc into the lower dimension space to obtain an embedded concatenated dictionary Dc, wherein the classifying a MDDP or NAMDDP as an anomaly includes classifying the MDDP or NAMDDP as an anomaly based on embedded concatenated dictionary Dc,

whereby the repeating step (b) p≥1 times removes randomization effects related to the randomized LU decomposition by causing eigenvalues of each dictionary D to decline, thereby providing a more robust embedded concatenated dictionary Dc and improving anomaly detection performance.

11. The computer program product of claim 10 , wherein the applying randomized LU decomposition to obtain a dictionary D includes applying randomized LU to matrix A to obtain two matrices L and P and constructing dictionary D from the multiplication of matrix L with a transpose of matrix P.

12. The computer program product of claim 10 , wherein the applying of randomized LU decomposition to obtain a dictionary D and the embedding of dictionary D in a lower dimension embedded space include applying the randomized LU decomposition and embedding dictionary D by dynamically configuring computer system parameters on-the-fly, whereby the dynamic on-the-fly configuring of computer system parameters is done without changing computer system software code and without halting computer system operation, thereby significantly improving computer system performance.

13. The computer program product of claim 10 , wherein the classifying a MDDP or a newly arrived MDDP (NAMDDP) as an anomaly includes classifying includes classifying without relying on a signature and/or a rule and/or on domain expertise.

14. The computer program product of claim 9 , wherein the applying randomized LU decomposition to obtain a dictionary D includes applying randomized LU to matrix A to obtain two matrices L and P and constructing dictionary D from the multiplication of matrix L with a transpose of matrix P.

15. The computer program product of claim 9 , wherein the applying of randomized LU decomposition to obtain a dictionary D and the embedding of dictionary D in a lower dimension embedded space include applying the randomized LU decomposition and embedding dictionary D by dynamically configuring computer system parameters on-the-fly, whereby the dynamic on-the-fly configuring of computer system parameters is done without changing computer system software code and without halting computer system operation, thereby significantly improving computer system performance.

16. The computer program product of claim 9 , wherein the classifying a MDDP or a newly arrived MDDP (NAMDDP) as an anomaly includes classifying without relying on a signature and/or a rule and/or on domain expertise.

17. A computer system, comprising a hardware processor configurable to perform a method comprising steps of:

a) receiving data comprising a plurality of measurements m of multidimensional data points (MDDPs), each data point having n features wherein n>3, wherein m>>n and wherein the data forms a matrix A=m×n;

b) applying randomized lower and upper triangular matrix (LU) decomposition to matrix A to obtain a dictionary D in the form of a matrix k×n where k is the rank of matrix A and k<<m;

c) embedding dictionary D in a lower dimension space that has a dimension lower than n, thereby obtaining an embedded dictionary D;

d) in the lower dimension space and based on embedded dictionary D, classifying a MDDP or a newly arrived MDDP (NAMDDP) as an anomaly;

e) preparing an alert related to the anomaly; and

f) sending the alert to a data client system,

whereby the combined reduction in the size of both m and n reduces computer processing requirements, thereby enhancing anomaly detection performance.

18. The computer system of claim 17 , wherein the method further comprises:

g) repeating step (b) p≥1 times to obtain p additional dictionaries D(p);

h) concatenating each additional dictionary D(p) with all previously obtained dictionaries into a concatenated dictionary Dc; and

i) embedding concatenated dictionary Dc into the lower dimension space to obtain an embedded concatenated dictionary Dc, wherein the classifying a MDDP or NAMDDP as an anomaly includes classifying the MDDP or NAMDDP as an anomaly based on embedded concatenated dictionary Dc,

whereby the repeating step (b) p≥1 times removes randomization effects related to the randomized LU decomposition by causing eigenvalues of each dictionary D to decline, thereby providing a more robust embedded concatenated dictionary Dc and improving anomaly detection performance.

19. The computer system of claim 17 , wherein the applying randomized LU decomposition to obtain a dictionary D includes applying randomized LU to matrix A to obtain two matrices L and P and constructing dictionary D from the multiplication of matrix L with a transpose of matrix P.

20. The computer system of claim 17 , wherein the applying of randomized LU decomposition to obtain a dictionary D and the embedding of dictionary D in a lower dimension embedded space include applying the randomized LU decomposition and embedding dictionary D by dynamically configuring computer system parameters on-the-fly, whereby the dynamic on-the-fly configuring of computer system parameters is done without changing computer system software code and without halting computer system operation, thereby significantly improving computer system performance.