Anomaly detection based on communication between entities over a network
A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A method comprising:
receiving, by a computer system, event data associated with a communication between an internal entity within a computer network and an external entity outside the computer network, the event data including an identifier associated with a particular entity, wherein the particular entity is the internal entity or the external entity;
analyzing, by the computer system, a plurality of characters in the identifier by processing the event data;
assigning, by the computer system, a feature score based on the analysis, wherein the feature score is indicative of a level of confidence that the identifier is machine generated and a level of randomness in a sequence of characters in the identifier;
determining, by the computer system, that the level of randomness satisfies a specified criterion; and
detecting, by the computer system, an anomaly based on the feature score, wherein the anomaly is detected in response to determining that the level of randomness satisfies the specified criterion.
2. The method of claim 1 , wherein the detected anomaly is indicative of malware communications.
3. The method of claim 1 , wherein the feature score is representative of a quantified evaluation of risk associated with the particular entity.
4. The method of claim 1 , wherein analyzing the plurality of characters in the identifier includes:
analyzing a sequencing of the plurality of characters in the identifier.
5. The method of claim 1 , wherein analyzing the plurality of characters in the identifier includes:
performing an n-gram analysis of the identifier to determine a level of randomness in a sequence of characters in the identifier.
6. The method of claim 1 , wherein analyzing the plurality of characters in the identifier includes:
performing a lexical analysis of the identifier using natural language processing.
7. The method of claim 1 , further comprising:
assigning, by the computer system, a second feature score based on any of:
timing of communications associated with the particular entity;
sequencing of communications associated with the particular entity;
data transmission statistics associated with the particular entity; or
referral strings associated with the particular entity;
wherein the anomaly is further detected based on the second feature score.
8. The method of claim 1 , wherein detecting the anomaly includes:
assigning an anomaly score based on the feature score; and
determining that the anomaly score satisfies a specified criterion;
wherein the anomaly is detected in response to determining that the anomaly score satisfies the specified criterion.
9. The method of claim 1 , wherein detecting the anomaly includes:
assigning an anomaly score based on a weighted combination of the feature score and one or more other feature scores associated with the particular entity; and
determining that the anomaly score satisfies a specified criterion;
wherein the anomaly is detected in response to determining that the anomaly score satisfies the specified criterion.
10. The method of claim 1 , wherein assigning the feature score includes:
processing the event data using a machine learning model, the machine learning model including:
model processing logic defining a process for assigning the feature score based on the event data; and
a model state defining a set of parameters for applying the model processing logic.
11. The method of claim 1 , wherein detecting the anomaly includes:
processing the feature score using an anomaly model, the anomaly model including:
model processing logic defining a process for assigning an anomaly score based on the feature score; and
a model state defining a set of parameters for applying the model processing logic; and
assigning an anomaly score based on the processing of the feature score; and
determining that the anomaly score satisfies a specified criterion;
wherein the anomaly is detected in response to determining that the anomaly score satisfies the specified criterion.
12. The method of claim 1 , wherein detecting the anomaly includes:
determining a volume of event data associated with the communication between the internal entity and the external entity;
processing the feature score using:
a first anomaly model if the volume of event data is at or above a threshold volume; or
a second anomaly model if the volume of event data is below the threshold volume.
13. The method of claim 1 , wherein detecting the anomaly includes:
processing the feature score using a plurality of machine-learning models;
assigning a plurality of intermediate anomaly scores, each of the plurality of intermediate anomaly scores based on processing of the feature score according to one of the plurality of machine-learning models;
processing the plurality of intermediate anomaly scores according to an ensemble-learning model;
assigning an anomaly score based on processing the plurality of intermediate anomaly scores; and
determining that the anomaly score satisfies a specified criterion;
wherein the anomaly is detected in response to determining that the anomaly score satisfies a specified criterion.
14. The method of claim 1 , further comprising:
annotating, by the computer system, the detected anomaly with data from an external data source external to the computer network.
15. The method of claim 1 , further comprising:
outputting, by the computer system, via a user interface, an indication of the detected anomaly to a user.
16. The method of claim 1 , further comprising:
outputting, by the computer system, via a user interface, in response to detecting the anomaly, an incident response output, the incident response output including:
the identifier associated with the external or internal entity;
the feature score; and
a recommended response based on the feature score.
17. The method of claim 1 , further comprising:
Incorporating, by the computer system, the detected anomaly into a network security graph, the network security graph including:
a plurality of nodes representing entities associated with the computer network, the entities including the particular entity; and
a plurality of edges, each of the plurality of edges linking two of the plurality of nodes and representing an association between the entities represented by the nodes;
wherein the detected anomaly is incorporated as a node linked to the particular entity by an edge.
18. The method of claim 1 , wherein detecting the anomaly is performed in real time as the event data are received.
19. The method of claim 1 , wherein detecting the anomaly is performed using Apache Storm or Apache Spark Streaming as a processing engine.
20. The method of claim 1 , wherein receiving the event data includes:
adaptively filtering the event data according to a dynamic whitelist.
21. The method of claim 1 , wherein the event data are receiving received from a plurality of entities associated with the computer network via an extract, transform, and load (ETL) pipeline.
22. The method of claim 1 , wherein the event data are timestamped machine data.
23. The method of claim 1 , wherein the event data include one or more of: domain name system (DNS) generated log data, firewall generated log data, or proxy generated log data.
24. The method of claim 1 , further comprising:
storing, by the computer system, anomaly data indicative of the detected anomaly in an anomaly graph data structure that includes:
a plurality of nodes, each of the plurality of nodes representing entities associated with the computer network, the entities including users and/or devices; and
a plurality of edges, each of the plurality of edges representing an anomaly linking two of the plurality of nodes.
25. The method of claim 1 , wherein the feature score is one of a plurality of feature scores included in an entity profile associated with the particular entity, and wherein detecting the anomaly includes processing the entity profile using an anomaly model.
26. The method of claim 1 , further comprising:
generating, by the computer system, an entity profile for the particular entity, the entity profile including a plurality of feature scores, the plurality of feature scores including the feature score assigned based on the analysis of the plurality of characters in the identifier, each of the plurality of feature scores assigned based on a different one of a plurality of different analyses of the event data;
wherein detecting the anomaly includes processing the entity profile using an anomaly model.
27. The method of claim 1 , wherein the identifier is any of: a domain name, a uniform resource locater (URL), uniform resource identifier (URI), an Internet Protocol (IP) address, a unique identifier (UID), a device identification, or a user identification.
28. A system comprising:
a processor; and
a memory unit having instructions stored thereon, which when executed by the processor cause the system to:
receive event data associated with a communication between an internal entity within a computer network and an external entity outside the computer network, the event data including an identifier associated with a particular entity, wherein the particular entity is the internal entity or the external entity;
analyze a plurality of characters in the identifier by processing the event data;
assign a feature score based on the analysis, wherein the feature score is indicative of a level of confidence that the identifier is machine generated and a level of randomness in a sequence of characters in the identifier;
determine that the level of randomness satisfies a specified criterion; and
detect an anomaly based on the feature score, wherein the anomaly is detected in response to determining that the level of randomness satisfies the specified criterion.
29. A non-transient computer readable medium containing instructions, execution of which by a computer system cause the computer system to:
receive event data associated with a communication between an internal entity within a computer network and an external entity outside the computer network, the event data including an identifier associated with a particular entity, wherein the particular entity is the internal entity or the external entity;
analyze a plurality of characters in the identifier by processing the event data;
assign a feature score based on the analysis, wherein the feature score is indicative of a level of confidence that the identifier is machine generated and a level of randomness in a sequence of characters in the identifier;
determine that the level of randomness satisfies a specified criterion; and
detect an anomaly based on the feature score, wherein the anomaly is detected in response to determining that the level of randomness satisfies the specified criterion.