Complex event processing of computer network data
View Patent ↗A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A method comprising:
transforming in real time raw event data, representing a sequence of events and captured from multiple sources in a computer network, into a stream of event feature sets without a known end-point to the stream, where the raw event data includes time-stamped machine data;
computing in real-time a score by processing a time slice of the stream of event feature sets through an active version of a machine learning model, wherein the time slice includes a most recent event feature set in the stream of event feature sets;
training, in parallel with said processing the time slice and responsive in real-time to said transforming the raw event data, a non-active version of the machine learning model with the time slice that is being processed through the active version for scoring, wherein the machine learning model is trained to represent a particular entity involved in a computer network activity represented by the raw event data;
identifying, by comparing the score against a threshold, a security-related anomaly or a security-related threat to enable remediation of the security-related anomaly or the security-related threat in the computer network as the stream of event feature sets is processed in real-time;
determining that the non-active version of the machine learning model is ready for active deployment based on at least one of: a number of event feature sets that have been used to train the non-active version, length of time that the non-active version has been in training, or whether a model state of the non-active version is converging; and
live-swapping in the non-active version as the active version to compute another score by processing a subsequent time slice from the stream of event feature sets through the live-swapped-in active version of the machine learning model.
2. The method of claim 1 , wherein the time slice is the most recent time slice received from the stream of event feature sets.
3. The method of claim 1 , wherein the time slice corresponds to an event in the sequence of events; and wherein said identifying the security-related anomaly or the security-related threat includes determining that the security-related anomaly or the security-related threat corresponds to the event.
4. The method of claim 1 , further comprising training the machine learning model by processing a previous time slice of the stream of event feature sets, prior to said computing the score.
5. The method of claim 1 , wherein said processing the time slice through the active version of the machine learning model includes processing the time slice through a model deliberation process logic configured by the active version of the machine learning model; wherein said training the non-active version of the machine learning model includes processing the time slice through a model training process logic; and wherein the model training process logic and the model deliberation process logic correspond to a particular machine learning model type.
6. The method of claim 1 ,
wherein
said live swapping includes re-configuring, without first terminating, a model deliberation process thread that is performing said computing in real-time.
7. The method of claim 1 , wherein the machine learning model is an unsupervised machine learning model.
8. The method of claim 1 , furthering comprising processing the stream of event feature sets through a plurality of machine learning models of different types to detect security-related anomalies or threats of different types.
9. The method of claim 1 , wherein the machine learning model is a supervised machine learning model or a semi-supervised machine learning model.
10. The method of claim 1 , wherein the machine learning model is a deep machine learning model.
11. The method of claim 1 , wherein the machine learning model is specific to a particular entity involved in the sequence of events, wherein the entity is a user, a device, a system, a network resource locator, an application, a process thread, or any combination thereof.
12. The method of claim 1 , wherein the machine learning model performs behavioral analysis of a particular entity involved in the sequence of events.
13. The method of claim 1 , wherein the machine learning model performs peer group analysis amongst entities involved in the sequence of events.
14. The method of claim 1 , wherein the machine learning model performs time series analysis on a sequence of the event feature sets.
15. The method of claim 1 , wherein the machine learning model performs graph correlation analysis amongst entities involved in the sequence of events.
16. The method of claim 1 , wherein the machine learning model includes a state machine specific to an entity involved in an event represented in the time slice.
17. The method of claim 1 , wherein said computing the score includes processing the stream of event feature sets through the machine learning model according to model deliberation processing logic specified by a model type associated with the machine learning model.
18. The method of claim 1 , further comprising training the machine learning model according to model training processing logic specified by a model type associated with the machine learning model.
19. The method of claim 1 , further comprising incrementally updating the machine learning model in real-time using the time slice of the event feature sets, wherein said incremental updating includes:
isolating a portion of a model state representative of the machine learning model affected by the event feature sets; and
re-training only the portion of the model state.
20. The method of claim 1 , further comprising simultaneously training multiple machine learning models, associated with different purposes or different entities, in real-time.
21. The method of claim 1 , wherein computing the score is performed in a distributed computation system implementing a task-parallel distributed data processing engine.
22. The method of claim 1 , further comprising training the machine learning model in a distributed computation system implementing a task-parallel distributed data processing engine.
23. The method of claim 1 , further comprising training the machine learning model in real-time using single-pass training processing logic.
24. The method of claim 1 , further comprising:
training the machine learning model; and
storing a model state of the machine learning model, resulting from said training, in a distributed cache for use in said computing of the score.
25. The method of claim 1 , further comprising outputting, to a user via a computer output device, a representation of the security-related anomaly or the security-related threat, in response to determining the security-related anomaly or the security-related threat.
26. The method of claim 1 , wherein the threshold is defined by the machine learning model.
27. A system comprising:
at least one hardware processor implementing an extract transform load (ETL) engine configured to transform in real time raw event data, representing a sequence of events and captured from multiple sources in a computer network, into an stream of event feature sets without a known end-point to the stream, where the raw event data includes time-stamped machine data;
at least one hardware processor implementing a model execution engine configured to compute in real-time a score by processing a time slice of the stream of event feature sets through an active version of a machine learning model and to train, in parallel with said processing the time slice and responsive in real-time to said transforming the raw event data, a non-active version of the machine learning model with the time slice that is being processed through the active version for scoring, wherein the machine learning model is trained to represent a particular entity involved in a computer network activity represented by the raw event data, and wherein the time slice includes a most recent event feature set in the stream of event feature sets; and
at least one hardware processor implementing a computing node configured to identify, by comparing the score against a threshold, a security-related anomaly or a security-related threat to enable remediation of the security-related anomaly or the security-related threat in the computer network as the stream of event feature sets is processed in real-time;
wherein the model execution engine is configured to determine that the non-active version of the machine learning model is ready for active deployment based on at least one of: a number of event feature sets that have been used to train the non-active version, length of time that the non-active version has been in training, or whether a model state of the non-active version is converging, and to live-swap in the non-active version as the active version to compute another score by processing a subsequent time slice from the stream of event feature sets through the live-swapped-in active version of the machine learning model.
28. A non-transitory computer readable medium storing instructions there on which, when executed by a processor, cause the processor to:
transform in real time raw event data, representing a sequence of events and captured from multiple sources in a computer network, into an stream of event feature sets without a known end-point to the stream, where the raw event data includes time-stamped machine data;
compute in real-time a score by processing a time slice of the stream of event feature sets through an active version of a machine learning model;
train, in parallel with said processing the time slice and responsive in real-time to said transforming the raw event data, a non-active version of the machine learning model with the time slice that is being processed through the active version for scoring, wherein the machine learning model is trained to represent a particular entity involved in a computer network activity represented by the raw event data and wherein the time slice includes a most recent event feature set in the stream of event feature sets;
identify, by comparing the score against a threshold, a security-related anomaly or a security-related threat to enable remediation of the security-related anomaly or the security-related threat in the computer network as the stream of event feature sets is processed in real-time;
determine that the non-active version of the machine learning model is ready for active deployment based on at least one of: a number of event feature sets that have been used to train the non-active version, length of time that the non-active version has been in training, or whether a model state of the non-active version is converging; and
live-swap in the non-active version as the active version to compute another score by processing a subsequent time slice from the stream of event feature sets through the live-swapped-in active version of the machine learning model.