System and method for anomaly detection in dynamically evolving data using random neural network decomposition
Detection systems, methods and computer program products comprising a non-transitory tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method for anomaly detection, a detected anomaly being indicative of an undesirable event. A detection system comprises a computer and an anomaly detection engine executable by the computer, the anomaly detection engine configured to perform a method comprising receiving data comprising a plurality m of multidimensional data points (MDDPs), each data point having n features, constructing a dictionary D based on the received data, embedding dictionary D into a lower dimension embedded space and classifying, based in the lower dimension embedded space, a MDDP as an anomaly or as normal.
1. A computer program product comprising: a non-transitory tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising:
a) receiving input data in the form of a matrix A of size m×n and rank k, wherein m represents a plurality of multidimensional data points (MDDPs) and wherein n represents a dimension of each MDDP;
b) constructing iteratively i dictionaries D i , wherein each D i is a matrix B i =A i ′R i of dimension k i ×n, wherein A i ′ is the transpose of matrix A i , wherein k i is a rank of matrix A i , wherein R i is a Gaussian distributed random matrix of dimension m i ×k i , wherein each m i is smaller than an iteration m i−1 , wherein m i−1 is an immediately preceding iteration, and wherein each Di is constructed by applying to A i a multi-layer feedforward artificial neural network made of one hidden layer (NN(A i )) such that ∥B i −NN(A i )∥ is minimized with respect to parameters of the neural network;
c) concatenating all dictionaries D i to construct a dictionary D; and
d) using dictionary D to classify a MDDP as an anomaly or as normal, wherein a detected anomaly is indicative of an undesirable event,
whereby a reduction in m enhances the performance of a computer including the computer program product in both processing and storage terms.
2. The computer program product of claim 1 , wherein the iterations are stopped when k i does not change between successive iterations.
3. The computer program product of claim 1 , wherein using dictionary D to classify the MDDP as an anomaly or as normal includes embedding dictionary D into a lower dimension embedded space to obtain a lower dimension embedded D and using the lower dimension embedded dictionary D to classify the MDDP as an anomaly or as normal.
4. The computer program product of claim 1 , wherein the classifying is performed automatically and/or unsupervised without relying on a signature and/or a rule and/or on domain expertise.
5. The computer program product of claim 1 , operative to perform the method offline for the MDDP or online for a newly arrived MDDP.
6. The computer program product of claim 1 , wherein the non-transitory tangible storage medium is cloud-based, hardware-server based and/or virtual-server based.
7. The computer program product of claim 1 , wherein parameter settings used by the instructions are dynamically, automatically and on-the-fly configurable while the instructions are executed by the processing circuit.
8. A computer system, comprising:
a) a preparation module configured to: receive input data in the form of a matrix A of size m×n and rank k, wherein m represents a plurality of multidimensional data points (MDDPs) and wherein n represents a dimension of each MDDP, construct iteratively i dictionaries D i , wherein each D i is a matrix B i =A i ′R i of dimension k i ×n, wherein A i ′ is the transpose of matrix A i , wherein k i is a rank of matrix A i , wherein R i is a Gaussian distributed random matrix of dimension m i ×k i , wherein each m, is smaller than an iteration wherein m i−1 is an immediately preceding iteration, and wherein each D i is constructed by applying to A i a multi-layer feedforward artificial neural network made of one hidden layer NN(A i ) such that ∥B i −NN(A i )∥ is minimized with respect to parameters of the neural network, and to concatenate all dictionaries D i to construct a dictionary D; and
b) an anomaly detection system including an anomaly detection engine configured to use dictionary D to classify a MDDP as an anomaly or as normal, wherein a detected anomaly is indicative of an undesirable event,
whereby a reduction in m enhances the performance of the computer system in both processing and storage terms.
9. The computer system of claim 8 , wherein the iterations are stopped when k does not change between successive iterations.
10. The computer system of claim 8 , wherein using dictionary D to classify the MDDP as an anomaly or as normal includes a configuration to embed dictionary D into a lower dimension embedded space to obtain a lower dimension embedded D and to use the lower dimension embedded dictionary D to classify the MDDP as an anomaly or as normal.
11. The computer system of claim 8 , wherein the anomaly detection engine is further configured to set a threshold, wherein the classifying of the MDDP as an anomaly or as normal is performed based on the embedded space and the set threshold.
12. The computer system of claim 8 , wherein the classifying is performed automatically and/or unsupervised without relying on a signature and/or a rule and/or on domain expertise.
13. The computer system of claim 8 , operative to perform the method offline for the MDDP or online for a newly arrived MDDP.
14. The computer system of claim 8 , wherein the computer system is cloud-based, hardware-server based and/or virtual-server based.
15. The computer system of claim 8 , wherein parameter settings used by the computer system are dynamically, automatically and on-the-fly configurable.
16. The computer system of claim 8 , wherein the preparation module is configured to receive the input data via a plurality of interfaces.
17. A method, comprising:
a) receiving input data in the form of a matrix A of size m×n and rank k, wherein m represents a plurality of multidimensional data points (MDDPs) and wherein n represents a dimension of each MDDP;
b) constructing iteratively i dictionaries D i , wherein each D i is a matrix B i =A i ′R i of dimension k i ×n, wherein A i ′ is the transpose of matrix A i , wherein k i is a rank of matrix A i , wherein R i is a Gaussian distributed random matrix of dimension m i ×k i , wherein each m i is smaller than an iteration m i−1 , wherein m i−1 is an immediately preceding iteration, and wherein each D i is constructed by applying to A i a multi-layer feedforward artificial neural network made of one hidden layer (NN(A i )) such that ∥B i −NN(A i )∥ is minimized with respect to parameters of the neural network;
c) concatenating all dictionaries D i to construct a dictionary D; and
d) using dictionary D to classify a MDDP as an anomaly or as normal, wherein a detected anomaly is indicative of an undesirable event,
whereby a reduction in m enhances the performance of a computer performing the method in both processing and storage terms.
18. The method of claim 17 , wherein the iterations are stopped when k i does not change between successive iterations.
19. The method of claim 17 , wherein using dictionary D to classify the MDDP as an anomaly or as normal includes embedding dictionary D into a lower dimension embedded space to obtain a lower dimension embedded D and using the lower dimension embedded dictionary D to classify the MDDP as an anomaly or as normal.
20. The method of claim 17 , wherein the classifying is performed automatically and/or unsupervised without relying on a signature and/or a rule and/or on domain expertise.