System and method for anomaly detection in dynamically evolving data using random neural network decomposition
Detection systems, methods and computer program products comprising a non-transitory tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method for anomaly detection, a detected anomaly being indicative of an undesirable event. A detection system comprises a computer and an anomaly detection engine executable by the computer, the anomaly detection engine configured to perform a method comprising receiving data comprising a plurality m of multidimensional data points (MDDPs), each data point having n features, constructing a dictionary D based on the received data, embedding dictionary D into a lower dimension embedded space and classifying, based in the lower dimension embedded space, a MDDP as an anomaly or as normal.
1. A computer program product comprising: a non-transitory tangible storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising:
a) receiving input data comprising a plurality of measurements m of multidimensional data points (MDDPs), each data point having n features;
b) applying random projection and neural network (RPNN) processing to at least a part of the input data to construct a dictionary D, thereby reducing m, wherein the at least part of the input data is in the form of a matrix A with rank k and wherein the applying RPNN processing to the at least a part of the input data to construct the dictionary D includes applying RPNN iteratively in a plurality of iterations i, each iteration i resulting in a new dictionary D i and a reduced rank matrix A i , and concatenating all new dictionaries D i to construct dictionary D;
c) embedding dictionary D into a lower dimension embedded space to obtain a lower dimension embedded D; and
d) classifying, based on the lower dimension embedded D, a MDDP as an anomaly or as normal, wherein a detected anomaly is indicative of an undesirable event,
whereby the reduction in m enhances the performance of a computer including the computer program product in both processing and storage terms.
2. The computer program product of claim 1 , further comprising: setting a threshold, wherein the classifying of a MDDP as an anomaly or as normal is performed based the embedded space and the set threshold.
3. The computer program product of claim 1 , wherein the classifying is performed automatically and/or unsupervised without relying on a signature and/or a rule and/or on domain expertise.
4. The computer program product of claim 1 , operative to perform the method offline for a MDDP or online for a newly arrived MDDP (NAMDDP).
5. The computer program product of claim 1 , wherein the non-transitory tangible storage medium is cloud-based, hardware-server based and/or virtual-server based.
6. The computer program product of claim 1 , wherein parameter settings used by the instructions are dynamically, automatically and on-the-fly configurable while the instructions are executed by the processing circuit.
7. The computer program product of claim 1 , wherein the data is received from a plurality of data sources.
8. The computer program product of claim 1 , wherein the data is received via a plurality of interfaces by a preparation module which performs the applying random projection and neural network (RPNN) processing to at least a part of the input data to construct a dictionary D.
9. A computer system, comprising:
a) a preparation module configured to receive input data comprising a plurality of measurements m of multidimensional data points (MDDPs), each data point having n features, the preparation module further configured to apply random projection and neural network (RPNN) processing to at least a part of the input data to construct a dictionary D thereby reducing m, whereby the reduction in m enhances the performance of the computer system in both processing and storage terms,
wherein at least part of the input data is in the form of a matrix A with rank k and wherein the configuration to apply RPNN processing to the at least a part of the input data to construct the dictionary D includes a configuration to apply RPNN iteratively in a plurality of iterations i, each iteration i resulting in a new dictionary Di and a reduced rank matrix Ai and concatenating all new dictionaries Di to construct dictionary D; and
b) an anomaly detection system including an anomaly detection engine configured to embed dictionary D into a lower dimension embedded space to obtain a lower dimension embedded D, and to classify, based on the lower dimension embedded D, a MDDP as an anomaly or as normal.
10. The computer system of claim 9 , wherein the anomaly detection engine is further configured to set a threshold, wherein the classifying of a MDDP as an anomaly or as normal is performed based on the embedded space and the set threshold.
11. The computer system of claim 9 , wherein the classifying is performed automatically and/or unsupervised without relying on a signature and/or a rule and/or on domain expertise.
12. The computer system of claim 9 , operative to perform the method offline for a MDDP or online for a newly arrived MDDP (NAMDDP).
13. The computer system of claim 9 , wherein the computer system is cloud-based, hardware-server based and/or virtual-server based.
14. The computer system of claim 9 , wherein parameter settings used by the computer system are dynamically, automatically and on-the-fly configurable.
15. The computer system of claim 9 , wherein the preparation module is configured to receive the input data from a plurality of data sources.
16. The computer system of claim 9 , wherein the preparation module is configured to receive the input data is received via a plurality of interfaces.
17. A method comprising:
a) receiving input data comprising a plurality of measurements m of multidimensional data points (MDDPs), each data point having n features;
b) applying random projection and neural network (RPNN) processing to at least a part of the input data to construct a dictionary D, thereby reducing m, wherein the at least part of the input data is in the form of a matrix A with rank k and wherein the applying RPNN processing to the at least a part of the input data to construct the dictionary D includes applying RPNN iteratively in a plurality of iterations i, each iteration i resulting in a new dictionary D i and a reduced rank matrix A i , and concatenating all new dictionaries D i to construct dictionary D;
c) embedding dictionary D into a lower dimension embedded space to obtain a lower dimension embedded D; and
d) classifying, based on the lower dimension embedded D, a MDDP as an anomaly or as normal, wherein a detected anomaly being indicative of an undesirable event,
whereby the reduction in m enhances the performance of a computer performing the method in both processing and storage terms.