ANOMALY DETECTION BASED ON ENSEMBLE MACHINE LEARNING MODEL
A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1 . (canceled)
2 . A method comprising:
determining a volume of event data used to generate an entity profile associated with an entity, the entity profile including a plurality of feature scores;
identifying an anomaly model to use to process the entity profile based on the determined volume of event data used to generate the entity profile, wherein
a first anomaly model is used to process the entity profile when the volume of event data exceeds a threshold volume, and
a second anomaly model is used to process the entity profile when the volume of event data is below the threshold volume;
processing the entity profile in accordance with the identified anomaly model; and
generating an anomaly score based on the processing of the entity profile in accordance with the identified anomaly model.
3 . The method of claim 2 , wherein feature scores of the plurality of feature scores are generated by analysis of the event data.
4 . The method of claim 2 , wherein feature scores of the plurality of feature scores are generated based on a timing analysis of the event data, a lexical analysis of the event data, a communications statistics of the event data, a sequencing analysis of the event data, an entity associations analysis of the event data, and/or a referral analysis of the event data.
5 . The method of claim 2 , wherein the first anomaly model comprises an ensemble learning model.
6 . The method of claim 2 , wherein the second anomaly model comprises a weighted linear combination.
7 . The method of claim 2 , wherein the first anomaly model comprises an ensemble learning model, and the second anomaly model comprises a weighted linear combination.
8 . The method of claim 2 , wherein the anomaly threshold is a static threshold.
9 . The method of claim 2 , wherein the anomaly threshold is a dynamic threshold that adaptively changes based on at least one of an overall volume of event data being generated on a computer network, a type of entity to which the anomaly score is applied, a set of user configuration preference, and a set of types of analysis used to generate the plurality of feature scores.
10 . The method of claim 2 , further comprising detecting an anomaly in response to determining that the anomaly score satisfies a specified criterion.
11 . The method of claim 2 , further comprising receiving the event data associated with the entity on a computer network.
12 . The method of claim 2 , further comprising:
detecting an anomaly in response to determining that the anomaly score satisfies a specified criterion; and
outputting an indication of the detected anomaly for displaying to a user.
13 . The method of claim 2 , wherein the event data are timestamped machine data.
14 . The method of claim 2 , wherein the event data include one or more of domain name system (DNS) generated log data, firewall generated low data, or proxy generated log data.
15 . The method of claim 2 , wherein the event data includes an identifier associated with the entity, and wherein at least one feature score of the plurality of feature scores is indicative of a level of confidence that the identifier is machine generated.
16 . The method of claim 2 , wherein the event data is associated with a communication between an internal entity within a computer network and an external entity outside the computer network.
17 . A system comprising:
a processor; and
a memory having instructions stored therein, execution of which by the processor causes the system to:
determine a volume of event data used to generate an entity profile associated with an entity, the entity profile including a plurality of feature scores;
identify an anomaly model to use to process the entity profile based on the determined volume of event data used to generate the entity profile, wherein
a first anomaly model is used to process the entity profile when the volume of event data exceeds a threshold volume, and
a second anomaly model is used to process the entity profile when the volume of event data is below the threshold volume;
process the entity profile in accordance with the identified anomaly model; and
generate an anomaly score based on the processing of the entity profile in accordance with the identified anomaly model.
18 . The system of claim 17 , wherein the first anomaly model comprises an ensemble learning model, and the second anomaly model comprises a weighted linear combination.
19 . The system of claim 17 , wherein the anomaly threshold is a dynamic threshold that adaptively changes based on at least one of an overall volume of event data being generated on a computer network, a type of entity to which the anomaly score is applied, a set of user configuration preference, and a set of types of analysis used to generate the plurality of feature scores.
20 . A non-transitory machine-readable storage medium containing instructions, execution of which by a computer system causes the computer system to perform operations comprising:
determining a volume of event data used to generate an entity profile associated with an entity, the entity profile including a plurality of feature scores;
identifying an anomaly model to use to process the entity profile based on the determined volume of event data used to generate the entity profile, wherein
a first anomaly model is used to process the entity profile when the volume of event data exceeds a threshold volume, and
a second anomaly model is used to process the entity profile when the volume of event data is below the threshold volume;
processing the entity profile in accordance with the identified anomaly model; and
generating an anomaly score based on the processing of the entity profile in accordance with the identified anomaly model.
21 . The non-transitory machine-readable storage medium of claim 20 , wherein the first anomaly model comprises an ensemble learning model, and the second anomaly model comprises a weighted linear combination.