Anomaly detection based on ensemble machine learning model
A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A method comprising:
receiving, by a computer system, event data associated with an entity on a computer network;
analyzing, by the computer system, the event data;
generating, by the computer system, a plurality of feature scores for the entity based on a result of analyzing the event data;
creating, by the computer system, an entity profile uniquely associated with the entity, the entity profile including the plurality of feature scores for the entity;
accessing the entity profile to read the plurality of features scores for the entity;
processing the plurality of feature scores for the entity, accessed from the entity profile, by using a plurality of machine-learning models;
generating a plurality of intermediate anomaly scores for the entity, each based on processing of a respective one of the plurality of feature scores of the entity using a respective one of the plurality of machine-learning models;
processing the plurality of intermediate anomaly scores for the entity according to an ensemble learning model;
generating an anomaly score for the entity based on processing the plurality of intermediate anomaly scores for the entity according to the ensemble learning model; and
detecting an anomaly associated with the entity in response to determining that the anomaly score for the entity satisfies a specified criterion.
2. The method of claim 1 , wherein the detected anomaly is indicative of a malware communication.
3. The method of claim 1 , wherein each of the plurality of feature scores is representative of a quantified evaluation of risk associated with the entity.
4. The method of claim 1 , wherein detecting the anomaly includes assigning the anomaly score based on a weighted combination of the plurality of feature scores.
5. The method of claim 1 , wherein generating a feature score of the plurality of feature scores includes:
processing the event data using a machine learning model, the machine learning model including:
model processing logic defining a process for assigning the feature score based on the event data; and
a model state defining a set of parameters for applying the model processing logic;
wherein the anomaly is detected in response to determining that the anomaly score satisfies the specified criterion.
6. The method of claim 1 , wherein detecting the anomaly includes:
determining a volume of event data associated with a communication between the entity and another entity;
using the ensemble-learning model if the volume of event data is determined to be at or above a threshold volume.
7. The method of claim 1 , wherein the event data is associated with a communication between an internal entity within a computer network and an external entity outside the computer network.
8. The method of claim 1 , wherein the event data includes an identifier associated with the entity, and wherein at least one feature score of the plurality of feature scores is indicative of a level of confidence that the identifier is machine generated.
9. The method of claim 1 , further comprising:
annotating, by the computer system, the detected anomaly with data from an external data source external to the computer network.
10. The method of claim 1 , further comprising:
outputting, by the computer system, via a user interface, an indication of the detected anomaly to a user.
11. The method of claim 1 , wherein the event data is timestamped machine data.
12. The method of claim 1 , wherein the event data include one or more of: domain name system (DNS) generated log data, firewall generated log data, or proxy generated log data.
13. The method of claim 1 , wherein detecting the anomaly includes processing the entity profile using an anomaly model.
14. A system comprising:
a processor; and
a memory having instructions stored therein, execution of which by the processor causes the system to:
receive event data associated with an entity on a computer network;
analyze the event data;
generate a plurality of feature scores for the entity based on a result of analyzing the event data;
creating, by the computer system, an entity profile uniquely associated with the entity, the entity profile including the plurality of feature scores for the entity;
accessing the entity profile to read the plurality of features scores for the entity;
processing the plurality of feature scores for the entity, accessed from the entity profile, by using a plurality of machine-learning models;
generating a plurality of intermediate anomaly scores for the entity, each based on processing of a respective one of the plurality of feature scores for the entity using a respective one of the plurality of machine-learning models;
processing the plurality of intermediate anomaly scores for the entity according to an ensemble-learning model;
generating an anomaly score for the entity based on processing the plurality of intermediate anomaly scores for the entity according to the ensemble-learning model; and
detecting an anomaly associated with the entity in response to determining that the anomaly score for the entity satisfies a specified criterion.
15. The system of claim 14 , wherein the detected anomaly is indicative of a malware communication.
16. The system of claim 14 , wherein each of the plurality of feature scores is representative of a quantified evaluation of risk associated with the particular entity.
17. A non-transitory machine-readable storage medium containing instructions, execution of which by a computer system causes the computer system to perform operations comprising:
receiving event data associated with an entity on a computer network;
analyzing the event data;
generating a plurality of feature scores for the entity based on a result of analyzing the event data;
creating, by the computer system, an entity profile uniquely associated with the entity, the entity profile including the plurality of feature scores for the entity;
accessing the entity profile to read the plurality of features scores for the entity;
processing the plurality of feature scores for the entity, accessed from the entity profile, by using a plurality of machine-learning models;
generating a plurality of intermediate anomaly scores for the entity, each based on processing of a respective one of the plurality of feature scores for the entity using a respective one of the plurality of machine-learning models;
processing the plurality of intermediate anomaly scores for the entity according to an ensemble-learning model;
generating an anomaly score for the entity based on processing the plurality of intermediate anomaly scores for the entity according to the ensemble-learning model; and
detecting an anomaly associated with the entity in response to determining that the anomaly score for the entity satisfies a specified criterion.
18. The non-transitory machine-readable storage medium of claim 17 , such that detecting the anomaly includes assigning the anomaly score based on a weighted combination of the plurality of feature scores.