Sharing of machine learning model state between batch and real-time processing paths for detection of network security issues
A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A method comprising:
implementing, in a computer system, at least one of i) a batch event processing engine configured to process a batch of stored event data, or ii) a real-time event processing engine configured to process an unbounded stream of event data, the batch event processing engine and the real-time event processing engine each being configured, or having previously been configured, to use a particular machine-learning model to detect network security related issues, wherein use of the particular machine-learning model by a first one of the batch event processing engine or the real-time event processing engine on a first set of data results in, or previously resulted in, training the particular machine-learning model to a particular model state; and
causing the other one of the batch event processing engine or the real-time event processing engine to use the particular machine-learning model, trained to the particular model state by the first one of the batch event processing engine or the real-time event processing engine, to detect network security related issues from a second set of data.
2. The method of claim 1 , further comprising retrieving at least a portion of the stored event data by sending, to a datastore containing the event data, a query or a command to be executed by the datastore.
3. The method of claim 1 , further comprising retrieving at least a portion of the stored event data by specifying, in a query or a command to be executed by a datastore, a manner in which resulting data from the datastore is to be delivered.
4. The method of claim 1 , further comprising retrieving at least a portion of the stored event data by specifying, in a query or a command to be executed by a datastore, a manner in which resulting data from the datastore is to be delivered,
wherein the manner includes selecting a particular set of the resulting data as the specific event data.
5. The method of claim 1 , further comprising retrieving at least a portion of the stored event data by specifying, in a query or a command to be executed by a datastore, a manner in which resulting data from the datastore is to be delivered,
wherein the manner includes specifying a time range of the resulting data.
6. The method of claim 1 , further comprising retrieving at least a portion of the stored event data by specifying, in a query or a command to be executed by a datastore, a manner in which resulting data from the datastore is to be delivered,
wherein the manner includes specifying an order of a set of the resulting data.
7. The method of claim 1 , further comprising retrieving at least a portion of the stored event data by specifying, in a query or a command to be executed by a datastore, a manner in which resulting data from the datastore is to be delivered,
wherein the manner includes specifying an order of a set of the resulting data, and
wherein the order includes one or more of: event time, data format, or a type of event that particular resulting data represent.
8. The method of claim 1 , further comprising retrieving at least a portion of the stored event data by specifying, in a query or a command to be executed by a datastore, a manner in which resulting data from the datastore is to be delivered,
wherein the manner includes specifying an order of a set of the resulting data, and
wherein the order specifies that log files with device information are to be delivered first, followed by log files that associate user data with devices, followed by other files.
9. The method of claim 1 , further comprising retrieving at least a portion of the stored event data by specifying, in a query or a command to be executed by a datastore, a manner in which resulting data from the datastore is to be delivered, and
wherein the manner includes specifying an order of a set of the resulting data,
wherein the order specifies that DHCP logs are to be delivered first, followed by AD or VPN logs, followed by other files.
10. The method of claim 1 , further comprising retrieving at least a portion of the stored event data by specifying, in a query or a command to be executed by a datastore, a manner in which resulting data from the datastore is to be delivered, and
wherein the manner includes specifying that a particular type of the data is to be delivered on a higher priority than another particular type.
11. The method of claim 1 , further comprising:
retrieving at least a portion of the stored event data from a data store;
determining a priority of data type of data to be retrieved from the datastore; and
determining, based on the priority of the data type, a manner in which data from the datastore is to be delivered.
12. The method of claim 1 , further comprising:
retrieving at least a portion of the stored event data from a data store;
determining a priority of data type of data to be retrieved from the datastore; and
determining, based on the priority of the data type, a manner in which data from the datastore is to be delivered,
wherein the interaction with a datastore includes specifying, in a query or a command to be executed by the datastore, the manner in which data from the datastore is to be delivered.
13. The method of claim 1 , further comprising:
scheduling the batch event processing engine to process the specific event data by:
initiating a job for the batch event processing engine with the specific event data;
tracking a progress of the job; and
recording, based on the progress, an analytical result for events represented by the specific event data.
14. The method of claim 1 , further comprising:
determining a time schedule for performing interaction with a datastore to access at least a portion of the stored event data.
15. The method of claim 1 , further comprising:
receiving an initial result from a datastore regarding specific event data of the stored event data; and
determining, based on the initial result, a subsequent interaction with the datastore.
16. The method of claim 1 , further comprising:
receiving an initial result from a datastore regarding the specific event data of the stored event data; and
determining, based on the initial result, a subsequent interaction with the datastore,
wherein the subsequent interaction with the datastore includes causing the datastore to modify a delivery order of resulting data.
17. The method of claim 1 , interacting with a datastore that stores the stored event data, wherein the datastore is a Hadoop™ Distributed File System (HDFS) datastore.
18. The method of claim 1 , wherein the model state enables the batch event processing engine to use knowledge gained by the real-time event processing engine to discover a security-related issue in historic event data that is undetectable by the batch event processing engine without the knowledge.
19. The method of claim 1 , further comprising:
performing, by the batch event processing engine, an analysis of historic event data to detect a security-related issue, wherein the analysis includes at least one of: a lateral movement anomaly analysis, a behavioral peer analysis, a label propagation analysis, or a time-series anomaly analysis.
20. The method of claim 1 , further comprising:
locating, in a batch of historic event data, data representing a plurality of events associated with an entity; and
performing, by the batch event processing engine, a behavioral analysis of the entity to detect a behavioral anomaly.
21. The method of claim 1 , further comprising:
locating, in a batch of historic event data, data representing a plurality of events that are associated with behaviors performed by a plurality of entities; and
performing, by the batch event processing engine, anomaly analysis on the behaviors performed by a plurality of entities to detect a particular security-related anomaly.
22. The method of claim 1 , further comprising:
locating, by the batch event processing engine, a composite relationship graph associated with the historic event data; and
obtaining a projection of the composite relationship graph, based on a requirement of the particular machine-learning model.
23. The method of claim 1 , further comprising:
receiving user feedback regarding a determination of a detected security-related issue; and
updating the particular machine-learning model based on the user feedback.
24. The method of claim 1 , wherein the network security related issues include at least one of: a security-related anomaly or a security-related threat, wherein a security-related anomaly represents a detected fact, and wherein a security-related threat represents a security-related interpretation of one or more detected anomalies.
25. The method of claim 1 , wherein the event data of the stored event data and the event data of the unbounded stream of event data each comprise machine data.
26. The method of claim 1 , wherein the event data of the stored event data comprise timestamped machine data.
27. The method of claim 1 , wherein the particular machine-learning model is a reducible model.
28. The method of claim 1 , wherein the particular machine-learning model is a reducible model that is reducible in at least one of a training phase or a scoring phase.
29. A processing system comprising:
at least one processor and a memory accessible to the at least one processor, the memory containing code that configures the at least one processor to implement at least one of i) a batch event processing engine configured to process a batch of stored event data, or ii) a real-time event processing engine configured to process an unbounded stream of event data, the batch event processing engine and the real-time event processing engine each being configured, or having previously been configured, to use a particular machine-learning model to detect network security related issues, wherein use of the particular machine-learning model by a first one of the batch event processing engine or the real-time event processing engine on a first set of data results in, or previously resulted in, training the particular machine-learning model to a particular model state; and
where the system is configured to cause the other one of the batch event processing engine or the real-time event processing engine to use the particular machine-learning model, trained to the particular model state by the first one of the batch event processing engine or the real-time event processing engine, to detect network security related issues from a second set of data.
30. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:
Implementing at least one of i) a batch event processing engine configured to process a batch of stored event data, or ii) a real-time event processing engine configured to process an unbounded stream of event data, the batch event processing engine and the real-time event processing engine each being configured, or having previously been configured, to use a particular machine-learning model to detect network security related issues, wherein use of the particular machine-learning model by a first one of the batch event processing engine or the real-time event processing engine on a first set of data results in, or previously resulted in, training the particular machine-learning model to a particular model state; and
causing the other one of the batch event processing engine or the real-time event processing engine to use the particular machine-learning model, trained to the particular model state by the first one of the batch event processing engine or the real-time event processing engine, to detect network security related issues from a second set of data.