Data retrieval in security anomaly detection platform with shared model state between real-time and batch paths
A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A method comprising:
implementing a batch event processing engine on a distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data;
performing an interaction with a datastore to retrieve specific event data;
scheduling the batch event processing engine to process the specific event data; and
enabling the batch event processing engine to share a model state of a particular machine learning model, with a real-time event processing engine on the distributed data processing platform, the real-time event processing engine being configured to process an unbounded stream of event data, the particular machine learning model being configured to process a time slice of data for detecting a security-related issue,
wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, security-related knowledge gained from processing respective data.
2. The method of claim 1 , wherein the interaction with a datastore includes sending, to the datastore, a query or a command to be executed by the datastore.
3. The method of claim 1 , wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered.
4. The method of claim 1 , wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and
wherein the manner includes selecting a particular set of the resulting data as the specific event data.
5. The method of claim 1 , wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and
wherein the manner includes specifying a time range of the resulting data.
6. The method of claim 1 , wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and
wherein the manner includes specifying an order of a set of the resulting data.
7. The method of claim 1 , wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and
wherein the manner includes specifying an order of a set of the resulting data,
wherein the order includes one or more of: event time, data format, or a type of event that a particular resulting data represent.
8. The method of claim 1 , wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and
wherein the manner includes specifying an order of a set of the resulting data,
wherein the order specifies that log files with device information are to be delivered first, followed by log files that associate user data with devices, followed by other files.
9. The method of claim 1 , wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and
wherein the manner includes specifying an order of a set of the resulting data,
wherein the order specifies that DHCP logs are to be delivered first, followed by AD or VPN logs, followed by other files.
10. The method of claim 1 , wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, a manner in which resulting data from the datastore is to be delivered, and
wherein the manner includes specifying that a particular type of the resulting data is to be delivered on a higher priority than another particular type.
11. The method of claim 1 , further comprising:
determining a priority of data type in the resulting data from the datastore; and
determining, based on the priority of the data type, a manner in which resulting data from the datastore is to be delivered.
12. The method of claim 1 , further comprising:
determining a priority of data type in the resulting data from the datastore; and
determining, based on the priority of the data type, a manner in which resulting data from the datastore is to be delivered,
wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, the manner in which resulting data from the datastore is to be delivered.
13. The method of claim 1 , wherein said scheduling the batch event processing engine to process the specific event data includes:
initiating a job for the batch event processing engine with the specific event data;
tracking a progress of the job; and
recording, based on the progress, an analytical result for events represented by the specific event data.
14. The method of claim 1 , further comprising:
determining a time schedule for performing said interaction with the datastore.
15. The method of claim 1 , further comprising:
receiving an initial result from the datastore regarding the specific event data;
determining, based on the initial result, a subsequent interaction with the datastore.
16. The method of claim 1 , further comprising:
receiving an initial result from the datastore regarding the specific event data; and
determining, based on the initial result, a subsequent interaction with the datastore,
wherein the subsequent interaction with the datastore includes causing the datastore to modify a delivery order of resulting data.
17. The method of claim 1 , wherein the datastore is a Hadoop™ Distributed File System (HDFS) datastore.
18. The method of claim 1 , wherein the shared model state enables the batch event processing engine to use knowledge gained by the real-time event processing engine to discover a security-related issue in the historic event data that is undetectable by the batch event processing engine without the knowledge.
19. The method of claim 1 , further comprising:
performing, by the batch event processing engine, an analysis on the historic event data to detect a security-related issue, wherein the analysis includes at least one of: a lateral movement anomaly analysis, a behavioral peer analysis, a label propagation analysis, or a time-series anomaly analysis.
20. The method of claim 1 , further comprising:
locating, in the batch of historic event data, data representing a plurality of events associated with an entity; and
performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly.
21. The method of claim 1 , further comprising:
locating, in the batch of historic event data, data representing a plurality of events that are associated with behaviors performed by a plurality of entities; and
performing, by the batch event processing engine, anomaly analysis on the behaviors performed by a plurality of entities to detect a particular security-related anomaly.
22. The method of claim 1 , further comprising:
locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and
obtaining a projection of the composite relationship graph, based on a requirement of the particular machine learning model.
23. The method of claim 1 , further comprising:
receiving user feedback regarding a determination of a detected security-related issue; and
updating the particular machine learning model based on the user feedback.
24. The method of claim 1 , wherein the security-related issues include at least one of: a security-related anomaly or a security-related threat, wherein the security-related anomaly represents a detected fact, and wherein the security-related threat represents a security-related interpretation of one or more detected anomalies.
25. The method of claim 1 , wherein the event data comprise machine data.
26. The method of claim 1 , wherein the event data comprise timestamped machine data.
27. The method of claim 1 , wherein at least one of the event processing engines utilizes a particular machine learning model that is a reducible model.
28. The method of claim 1 , wherein at least one of the event processing engines utilizes a particular machine learning model that is a reducible model being reducible in at least one of: a training phase, or a scoring phase.
29. A computer system comprising:
a real-time event processing engine on the distributed data processing platform, the real-time event processing engine being configured to process an unbounded stream of event data; and
a batch event processing engine on a distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data;
wherein the system is configured to:
perform an interaction with a datastore to retrieve specific event data;
schedule the batch event processing engine to process the specific event data; and
enable the batch event processing engine to share a model state of a particular machine learning model, with the real-time event processing engine, the particular machine learning model being configured to process a time slice of data for detecting a security-related issue,
wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, security-related knowledge gained from processing respective data.
30. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:
implementing a batch event processing engine on a distributed data processing platform, wherein the batch event processing engine is configured to process a batch of historic event data;
performing an interaction with a datastore to retrieve specific event data;
scheduling the batch event processing engine to process the specific event data; and
enabling the batch event processing engine to share a model state of a particular machine learning model, with a real-time event processing engine on the distributed data processing platform, the real-time event processing engine being configured to process an unbounded stream of event data, the particular machine learning model being configured to process a time slice of data for detecting a security-related issue,
wherein the real-time event processing engine and the batch event processing engine each utilize the shared model state to share, with the other engine, security-related knowledge gained from processing respective data.