Distributed machine learning for anomaly detection
A tiered machine learning-based infrastructure comprises a first machine learning (ML) tier configured to execute within an enterprise network environment and that learns statistics for a set of use cases locally, and to alert deviations from the learned distributions. Use cases typically are independent from one another. A second machine learning tier executes external to the enterprise network environment and provides further learning support, e.g., by determining a correlation among multiple independent use cases that are running locally in the first tier. Preferably, the second tier executes in a cloud compute environment for scalability and performance.
1. A method for anomaly detection in association with an enterprise environment, comprising:
providing first machine learning to train at least a first and a second analytic, the first analytic corresponding to a first use case, and the second analytic corresponding to a second use case distinct from the first use case;
outputting to a second machine learning, and in a continuous manner, anomaly information derived from the first machine learning, the anomaly information including data points detected by the first machine learning as a result of applying the first and second analytics with respect to the first and second use cases; and
providing the second machine learning based on the anomaly information to capture a correlation among observed parameters in at least the first and second use cases;
wherein the first machine learning takes place in an enterprise network, and the second machine learning takes place in a cloud computing environment distinct from the enterprise network.
2. The method as described in claim 1 wherein each of the first and second use cases uses a distinct training data set.
3. The method as described in claim 1 wherein training data for at least one of the first and second use cases is time-series data.
4. The method as described in claim 1 further including outputting configuration and training data from the enterprise network to the cloud computing environment.
5. The method as described in claim 1 wherein the correlation is a multi-dimensional distance measure.
6. The method as described in claim 1 further including taking an action with respect to detected network activity or user behavior based the captured correlation provided by the second machine learning.
7. An apparatus, comprising:
hardware processors;
computer memory holding computer program instructions executed by the hardware processors for anomaly detection in association with an enterprise environment, the computer program instructions configured to:
provide first machine learning to train at least a first and a second analytic, the first analytic corresponding to a first use case, and the second analytic corresponding to a second use case distinct from the first use case;
output to a second machine learning, and in a continuous manner, anomaly information derived from the first machine learning, the anomaly information including data points detected by the first machine learning as a result of applying the first and second analytics with respect to the first and second use cases; and
provide the second machine learning based on the anomaly information to capture a correlation among observed parameters in at least the first and second use cases;
wherein the first machine learning takes place in an enterprise network, and the second machine learning takes place in a cloud computing environment distinct from the enterprise network.
8. The apparatus as described in claim 7 wherein each of the first and second use cases uses a distinct training data set.
9. The apparatus as described in claim 7 wherein training data for at least one of the first and second use cases is time-series data.
10. The apparatus as described in claim 7 wherein the computer program instructions are further configured to output configuration and training data from the enterprise network to the cloud computing environment.
11. The apparatus as described in claim 7 wherein the computer program instructions are configured to execute a multi-dimensional distance measure to determine the correlation.
12. The apparatus as described in claim 7 wherein the computer program instructions are further configured to take an action with respect to detected network activity or user behavior based the captured correlation provided by the second machine learning.
13. A computer program product in a non-transitory computer readable medium for use in first and second data processing systems for anomaly detection in association with an enterprise environment, the computer program product holding computer program instructions that, when executed by a respective one of the first and second data processing systems, are configured to:
provide first machine learning to train at least a first and a second analytic, the first analytic corresponding to a first use case, and the second analytic corresponding to a second use case distinct from the first use case;
output to a second machine learning, and in a continuous manner, anomaly information derived from the first machine learning, the anomaly information including data points detected by the first machine learning as a result of applying the first and second analytics with respect to the first and second use cases; and
provide the second machine learning based on the anomaly information to capture a correlation among observed parameters in at least the first and second use cases;
wherein the first machine learning takes place in the first data processing system within an enterprise network, and the second machine learning takes place in the second data processing system located in a cloud computing environment distinct from the enterprise network.
14. The computer program product as described in claim 13 wherein each of the first and second use cases uses a distinct training data set.
15. The computer program product as described in claim 13 wherein training data for at least one of the first and second use cases is time-series data.
16. The computer program product as described in claim 13 wherein the computer program instructions are further configured to output configuration and training data from the first data processing system to the second data processing system.
17. The computer program product as described in claim 13 wherein the computer program instructions are configured to execute a multi-dimensional distance measure to determine the correlation.
18. The computer program product as described in claim 13 wherein the computer program instructions are further configured to take an action with respect to detected network activity or user behavior based the captured correlation provided by the second machine learning.
19. A machine learning system for anomaly detection, comprising:
a memory;
a first machine learning system executing in a first operating environment, the first machine learning system training at least a first and a second analytic, the first analytic corresponding to a first use case, and the second analytic corresponding to a second use case distinct from the first use case, the first machine learning system training on the first and second use cases simultaneously; and
a second machine learning system executing in a second operating environment remote from the first machine learning system, the second machine learning system configured to capture a multi-dimensional distance measure correlation among observed parameters in at least the first and second use cases, the observed parameters being derived by the first machine learning system;
wherein the observed parameters are output from the first machine learning system to the second machine learning system in a continuous manner.
20. The machine learning system as described in claim 19 wherein the first machine learning system executes as an application in a Security Event and Incident Management Platform (SIEM), and wherein the second machine learning system executes as an application in a cloud compute infrastructure with the second operating environment.