System and method for detecting an undesirable event

1. A method for detecting an unknown undesirable event, comprising the steps of:

a) receiving a dataset I comprising a plurality n of multidimensional datapoints (MDDPs) with dimension m≥3, I being a matrix of size n×m and wherein n>>m;

normalizing the MDDPs to obtain distributions of normalized MDDPs (NMDDPs) and applying whitening principal component analysis (WPCA) to the distributions of NMDDPs to obtain a lower dimension embedded space with embedded normalized NMDDPs, wherein the applying of WPCA includes applying singular value decomposition (SVD) to a covariance matrix I T I where T is the transpose of matrix I, wherein the whitening in the WPCA renders the distributions of the NMDDPs less redundant and wherein covariance matrix I T I has a dimension m×m much smaller than n×m;

c) calculating distributions of distances D nn i , i=1, . . . , n of each embedded NMDDP from a plurality of nearest-neighbors (nn) to compute a threshold D nn t ; and

d) classifying a particular MDDP of the dataset or a newly arrived MDDP (NAMDDP) as an abnormal MDDP based on comparison with threshold D nn t , wherein the classification is automatic and unsupervised without relying on a signature, rules or domain expertise and wherein the particular MDDP classified as abnormal is indicative of the unknown undesirable event

whereby the whitening, the embedding in a lower dimension space and the application of SVD to covariance matrix I T I with a dimension much smaller than that of matrix I reduce computer memory needs and speed up computing operations.

2. The method of claim 1 , wherein the embedded space has a dimension k<m.

3. The method of claim 1 , wherein the step of calculating distributions of distances D nn i , i=1, . . . , n of each embedded NMDDP from a plurality of nearest-neighbors (nn) to compute a threshold D nn t includes applying a Gaussian mixture to each distribution to obtain Gaussian weights and using the Gaussian weights to compute threshold D nn t .

4. The method of claim 3 , wherein applying a Gaussian mixture to each distribution to obtain Gaussian weights and using the Gaussian weights to compute threshold D nn t includes computing threshold D nn t from a posterior probability for each element in D nn i .

5. The method of claim 1 , wherein the classification is performed offline.

6. The method of claim 1 , wherein the classification is performed online.

7. The method of claim 1 , wherein the unknown undesirable event is selected from the group consisting of a cyber-threat, a cyber-attack, an operational malfunction, an operational breakdown, a process malfunction, a process breakdown, a financial risk event, a financial threat event, a financial fraud event, money laundering and a financial network intrusion event.

8. The method of claim 1 , further comprising the step of scoring the particular abnormal MDDP.

9. A system for detecting an unknown undesirable event, comprising:

a) an input device configured to receive a dataset I comprising a plurality n of multidimensional datapoints (MDDPs) with dimension m≥3, I thereby being a matrix of size n×m;

b) a processor configured to normalize the MDDPs to obtain distributions of normalized MDDPs (NMDDPs) and to apply whitening principal component analysis (WPCA) to the distributions of NMDDPs to obtain a lower dimension embedded space with embedded normalized NMDDPs, wherein the applying of WPCA includes applying singular value decomposition (SVD) to a covariance matrix I T I where T is the transpose of matrix I, wherein the whitening in the WPCA renders the distributions of the NMDDPs less redundant and wherein covariance matrix I T I has a dimension m×m much smaller than n×m; and

c) a detection engine configured to calculate distributions of distances D nn i , i=1, . . . , n of each embedded NMDDP from a plurality of nearest-neighbors (nn) to compute a threshold D nn t and to classify a particular MDDP of the dataset or a newly arrived MDDP (NAMDDP) as an abnormal MDDP based on comparison with threshold D nn t , wherein the classification is automatic and unsupervised without relying on a signature, rules or domain expertise and wherein the particular MDDP classified as abnormal is indicative of the unknown undesirable event,

whereby the whitening, the embedding in a lower dimension space and the application of SVD to covariance matrix I T I with a dimension much smaller than that of matrix I reduce computer memory needs and speed up computing operations.

10. The method of claim 9 , wherein the configuration to calculate distributions of distances D nn i , i=1, . . . , n of each embedded NMDDP from a plurality of nearest-neighbors (nn) to compute a threshold D nn t includes a configuration to apply a Gaussian mixture to each distribution to obtain Gaussian weights and to use the Gaussian weights to compute threshold D nn t .

11. The method of claim 10 , wherein configuration to apply a Gaussian mixture to each distribution to obtain Gaussian weights and to use the Gaussian weights to compute threshold D nn t includes a configuration compute threshold D nn t from a posterior probability for each element in D nn i .

12. The method of claim 9 , wherein the classification is performed offline.

13. The method of claim 9 , wherein the classification is performed online.

14. The method of claim 9 , wherein the unknown undesirable event is selected from the group consisting of a cyber-threat, a cyber-attack, an operational malfunction, an operational breakdown, a process malfunction, a process breakdown, a financial risk event, a financial threat event, a financial fraud event, money laundering and a financial network intrusion event.