Sharing model state between real-time and batch paths in network security anomaly detection
View Patent ↗A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A method comprising:
implementing a real-time event processing engine on a distributed data processing platform, wherein the real-time event processing engine is configured to process an unbounded stream of event data to detect a plurality of network security-related issues and/or to train a machine learning model;
implementing a batch event processing engine on the distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data to detect a plurality of network security-related issues and/or to train a machine learning model; and
enabling the real-time event processing engine and the batch event processing engine to share a model state of a particular machine learning model, the particular machine learning model being configured to process a time slice of data to produce a score for detecting a network security-related issue,
wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, network security-related knowledge gained from processing one's respective data.
2. The method of claim 1 , wherein a size of the time slice is controlled by whichever one of the real-time processing engine and the batch processing engine utilizes the particular machine learning model.
3. The method of claim 1 , wherein the shared model state enables the batch event processing engine to use knowledge gained by the real-time event processing engine to discover a security-related issue in the historic event data.
4. The method of claim 1 , wherein the shared model state enables the batch event processing engine to use knowledge gained by the real-time event processing engine to discover a security-related issue in the historic event data that is undetectable by the batch event processing engine without the knowledge.
5. The method of claim 1 , wherein the shared model state enables the batch event processing engine to use new knowledge gained by the real-time event processing engine from processing the unbounded stream of event data, to inspect the historic event data to discover a security-related issue that would be undetectable by the batch event processing engine without the new knowledge,
wherein the new knowledge is knowledge gained after a last time the batch event processing engine performs historic event data inspection using the particular machine learning model.
6. The method of claim 1 , further comprising:
performing, by the batch event processing engine, an analysis on the historic event data to detect a security-related issue.
7. The method of claim 1 , further comprising:
performing, by the batch event processing engine, an analysis on the historic event data to detect a security-related issue, wherein the analysis includes at least one of: a lateral movement anomaly analysis, a behavioral peer analysis, a label propagation analysis, or a time-series anomaly analysis.
8. The method of claim 1 , further comprising:
performing, by the batch event processing engine, an analysis on the historic event data to detect a security-related issue, wherein the analysis results in acquisition of new knowledge of event data; and
updating, by the batch event processing engine, the shared model state of the particular machine learning model to incorporate the new knowledge;
and
performing, by the real-time event processing engine, an inspection of newly received event data based on the new knowledge, by using the shared model state.
9. The method of claim 1 , further comprising:
locating, in the batch of historic event data, data representing a plurality of events associated with an entity; and
performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly.
10. The method of claim 1 , further comprising:
locating, in the batch of historic event data, data representing a plurality of events that are associated with an entity; and
performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly, wherein said performing the behavioral analysis includes:
utilizing the particular machine learning model to compute a degree of behavioral deviation in behaviors within a given time slice as compared to a behavioral baseline specific to the entity; and
determining, based on the degree of behavioral deviation, whether the behavioral anomaly exists.
11. The method of claim 1 , further comprising:
locating, in the batch of historic event data, data representing a plurality of events that are associated with an entity; and
performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly, wherein said performing behavioral analysis includes:
utilizing the particular machine learning model to compute a degree of behavioral deviation in behaviors within a given time slice as compared to a behavioral baseline specific to the entity; and
determining, based on the degree of behavioral deviation, whether the behavioral anomaly exists,
wherein the particular machine learning model is trained by the real-time event processing engine.
12. The method of claim 1 , further comprising:
locating, in the batch of historic event data, data representing a plurality of events that are associated with an entity; and
performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly, wherein said performing behavioral analysis includes:
utilizing the particular machine learning model to compute a degree of behavioral deviation in behaviors within a given time slice as compared to a behavioral baseline specific to the entity; and
determining, based on the degree of behavioral deviation, whether the behavioral anomaly exists,
wherein the particular machine learning model is trained by the real-time event processing engine, and
wherein the behavioral baseline is established by the real-time event processing engine using the particular machine learning model.
13. The method of claim 1 , further comprising:
locating, in the batch of historic event data, data representing a plurality of events that are associated with behaviors performed by a plurality of entities; and
performing, by the batch event processing engine, anomaly analysis on the behaviors performed by a plurality of entities to detect a particular security-related anomaly.
14. The method of claim 1 , further comprising:
locating, in the batch of historic event data, data representing a plurality of events associated with behaviors performed by a plurality of entities; and
performing, by the batch event processing engine, anomaly analysis on the behaviors performed by a plurality of entities to detect a particular security-related anomaly,
wherein said anomaly analysis includes comparing a behavior of one entity to the behaviors of other entities in determining whether a behavioral deviation of the one entity represents the particular security-related anomaly.
15. The method of claim 1 , further comprising:
locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and
obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model.
16. The method of claim 1 , further comprising:
locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and
obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model,
wherein the composite relationship graph is based on information generated by the real-time event processing engine processing the unbounded stream of event data.
17. The method of claim 1 , further comprising:
locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and
obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model,
wherein the projection includes a graph of users associated with machines to facilitate tracking, by the particular machine learning model, of user lateral movement.
18. The method of claim 1 , further comprising:
locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and
obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model,
wherein the projection includes a graph that associates entities identified as having security-related issues to facilitate correlating, by the particular machine learning model, of user anomalies so as to identify sophisticated threats.
19. The method of claim 1 , further comprising:
locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and
obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model,
wherein the projection includes a graph of website visitation activities of users to facilitate identifying, by the particular machine learning model, of commonly accessed websites by potentially security-compromised users.
20. The method of claim 1 , further comprising:
receiving user feedback regarding a determination of a detected security-related issue; and
updating the particular machine learning model based on the user feedback.
21. The method of claim 1 , further comprising:
using a job controller to cause a retrieval of the historic event data; and
determining, by the job controller and based on a directory catalog, an order by which the batch event processing engine is to process the historic event data.
22. The method of claim 1 , wherein the security-related issues include at least one of: a security-related anomaly or a security-related threat, wherein the security-related anomaly represents a detected fact, and wherein the security-related threat represents a security-related interpretation of one or more detected anomalies.
23. The method of claim 1 , wherein the event data comprise machine data.
24. The method of claim 1 , wherein the event data comprise timestamped machine data.
25. The method of claim 1 , wherein the real-time event processing engine is implemented using Apache Storm™ or Apache Spark Streaming™.
26. The method of claim 1 , wherein the batch event processing engine is implemented using Apache Spark™.
27. The method of claim 1 , wherein the particular machine learning model is a reducible model.
28. The method of claim 1 , wherein the particular machine learning model is a reducible model being reducible in at least one of: a training phase, or a scoring phase.
29. A computer system comprising:
a real-time event processing engine implemented on a distributed data processing platform, wherein the real-time event processing engine is configured to process an unbounded stream of event data to detect a plurality of network security-related issues and/or to train a machine learning model;
a batch event processing engine implemented on the distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data to detect a plurality of network security-related issues and/or to train a machine learning model; and
wherein the real-time event processing engine and the batch event processing engine shares a model state of a particular machine learning model, the particular machine learning model being configured to process a time slice of data to produce a score for detecting a network security-related issue, and
wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, network security-related knowledge gained from processing one's respective data.
30. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:
implementing a real-time event processing engine on a distributed data processing platform, wherein the real-time event processing engine is configured to process an unbounded stream of event data to detect a plurality of network security-related issues and/or to train a machine learning model;
implementing a batch event processing engine on the distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data to detect a plurality of network security-related issues and/or to train a machine learning model; and
enabling the real-time event processing engine and the batch event processing engine to share a model state of a particular machine learning model, the particular machine learning model being configured to process a time slice of data to produce a score for detecting a network security-related issue,
wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, network security-related knowledge gained from processing one's respective data.