Malware communications detection
View Patent ↗A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A method comprising:
receiving, by a computer system, event data associated with a communication between an internal entity within a computer network and an external entity outside the computer network;
generating, by the computer system, a plurality of entity-specific feature scores by processing the event data, wherein each of the entity-specific features scores is representative of a quantified evaluation of a level of risk associated with a particular entity, each of the feature scores generated by a different one of a plurality of different types of analyses of the event data, the particular entity being the internal entity or the external entity;
wherein the plurality of entity-specific feature scores include:
a first entity-specific feature score based on a lexical analysis of an identifier associated with the particular entity; and
a second entity-specific feature score based on an analysis of the timing or sequencing of communications by the particular entity;
generating, by the computer system, an entity profile associated with the particular entity, the entity profile including the plurality of entity-specific feature scores;
generating, by the computer system, an anomaly score based on the entity profile; and
detecting, by the computer system, an anomaly if the anomaly score satisfies a specified criterion.
2. The method of claim 1 , wherein the detected anomaly is indicative of malware within the computer network.
3. The method of claim 1 , wherein the lexical analysis of the identifier associated with the particular entity includes any of
an analysis of the sequencing of characters in a domain name associated with the particular entity; or
an n-gram analysis of the domain name associated with the particular entity.
4. The method of claim 1 , wherein the plurality of different types of analyses of event data include analysis of one or more of the following:
sequencing of characters in a domain name associated with the particular entity;
timing of communications associated with the particular entity;
sequencing of communications associated with the particular entity;
data transmission statistics associated with the particular entity;
referral strings associated with the particular entity; or
entities within the computer network associated with the particular entity.
5. The method of claim 1 , wherein generating the first entity-specific feature score includes:
analyzing a sequencing of characters in a domain name associated with the particular entity; and
assigning the first entity-specific feature score based on the analysis;
wherein the first entity-specific feature score is indicative of a level of confidence that the domain name is machine-generated.
6. The method of claim 1 , wherein generating the first entity-specific feature score includes:
performing an n-gram analysis on a domain name associated with the particular entity; and
assigning the first entity-specific feature score based on the n-gram analysis;
wherein the first entity-specific feature score is indicative of a level of confidence that the domain name is machine-generated.
7. The method of claim 1 , wherein generating the second entity-specific feature score includes:
analyzing the timing of communications by the particular entity over a time period; and
assigning the second entity-specific feature score based on the analysis;
wherein the second entity-specific feature score is indicative of a level of confidence that the communications are machine-generated.
8. The method of claim 1 , wherein generating the second entity-specific feature score includes:
analyzing the timing of communications by the particular entity over a time period, the analyzing including calculating one or more of; a variance of interval times between communications and a periodicity of communications; and
assigning the second entity-specific feature score based on the analysis;
wherein the second entity-specific feature score is indicative of a level of confidence that the communications are machine-generated.
9. The method of claim 1 , wherein generating the second entity-specific feature score includes:
analyzing the sequencing of communications by the particular entity over a time period; and
assigning the second entity-specific feature score based on the analysis;
wherein the second entity-specific feature score is indicative of a level of confidence that the communications are associated with an exploit chain.
10. The method of claim 1 , wherein generating the plurality of entity-specific feature scores includes:
analyzing the data transmission statistics associated with the particular entity over a time period; and
assigning an entity-specific feature score based on the analysis;
wherein the entity-specific feature score is indicative of a level of confidence that the particular entity is associated with a command and control infrastructure external to the computer network.
11. The method of claim 1 , wherein generating the plurality of entity-specific feature scores includes:
processing the event data through a plurality of machine-learning models, each of the plurality of machine-learning models including:
model processing logic defining a process for assigning an entity-specific feature score based on the event data; and
a model state defining a set of parameters for applying the model processing logic; and
assigning the plurality of entity-specific features scores based on the processing of the event data;
wherein the entity profile is represented as a feature vector including the plurality of entity-specific feature scores.
12. The method of claim 1 , wherein generating the anomaly score includes:
processing the entity profile through an anomaly model, the anomaly model including:
model processing logic defining a process for assigning an anomaly score based on the plurality of entity-specific feature scores of the entity profile; and
a model state defining a set of parameters for applying the model processing logic; and
assigning the anomaly score based on the processing of the entity profile.
13. The method of claim 1 , wherein generating the anomaly score includes calculating a weighted combination of the plurality of entity-specific feature scores of the entity profile.
14. The method of claim 1 , wherein generating the anomaly score includes:
determining a volume of event data used to generate the entity profile;
processing the entity profile through:
a first anomaly model if the volume of event data is at or above a threshold volume; or
a second anomaly model if the volume of event data is below the threshold volume; and
assigning the anomaly score based on the processing of the entity profile.
15. The method of claim 1 , wherein generating the anomaly score includes:
processing the entity profile according to a plurality of machine-learning models;
assigning a plurality of intermediate anomaly scores, each of the plurality of intermediate anomaly scores based on processing of the entity profile according to one of the plurality of machine-learning models;
processing the plurality of intermediate anomaly scores according to an ensemble-learning model; and
assigning the anomaly score based on processing the plurality of intermediate anomaly scores.
16. The method of claim 1 , wherein generating the anomaly score includes:
comparing the identifier associated with the particular entity with entries in an external data source external to the computer network; and
assigning the anomaly score indicating a confidence level that the identifier matches a particular entry in the external data source based on the comparing.
17. The method of claim 1 , further comprising:
annotating, by the computer system, the detected anomaly with data from an external data source external to the computer network.
18. The method of claim 1 , further comprising:
outputting, via a user interface, an indication of the detected anomaly to a user.
19. The method of claim 1 , further comprising:
outputting, via a user interface, an incident response output based on the entity profile, the incident response output including:
the identifier associated with the particular entity;
the plurality of entity-specific feature scores; and
a recommended response based on the plurality of entity-specific feature scores.
20. The method of claim 1 , further comprising:
incorporating the detected anomaly into a network security graph, the network security graph including:
a plurality of nodes representing entities associated with the computer network, the entities including the particular entity; and
a plurality of edges, each of the plurality of edges linking two of the plurality of nodes and representing an association between the entities represented by the nodes;
wherein the detected anomaly is incorporated as a node linked to the particular entity by an edge.
21. The method of claim 1 , wherein detecting the anomaly is performed in real time as the event data are received.
22. The method of claim 1 , wherein detecting the anomaly is performed using Apache Storm or Apache Spark Streaming as a processing engine.
23. The method of claim 1 , wherein receiving the event data includes adaptively filtering the event data according to a dynamic whitelist.
24. The method of claim 1 , wherein receiving the event data includes receiving event data from a plurality of entities associated with the computer network via an extract, transform, and load (ETL) pipeline.
25. The method of claim 1 , wherein the event data are timestamped machine data.
26. The method of claim 1 , wherein the event data include one or more of: domain name system (DNS) generated log data, firewall generated log data, or proxy generated log data.
27. The method of claim 1 , further comprising storing the detected anomaly in an anomaly graph data structure that includes:
a plurality of nodes representing entities associated with the computer network, the entities including users and/or devices; and
a plurality of edges, each of the plurality of edges representing an anomaly linking two of the plurality of nodes.
28. The method of claim 1 , wherein the the identifier associated with the particular entity includes one or more of: a domain name, a uniform resource locator (URL), uniform resource identifier (URI), an Internet Protocol (IP) address, a unique identifier (UID), a device identification, or a user identification.
29. A system comprising:
a processor; and
a memory unit having instructions stored thereon, which when executed by the processor cause the system to:
receive event data associated with a communication between an internal entity within a computer network and an external entity outside the computer network;
generate a plurality of entity-specific feature scores by processing the event data, wherein each of the entity-specific features scores is representative of a quantified evaluation of a level of risk associated with a particular entity, each of the feature scores generated by a different one of a plurality of different types of analyses of the event data, the particular entity being the internal entity or the external entity;
wherein the plurality of entity-specific feature scores include:
a first entity-specific feature score based on a lexical analysis of an identifier associated with the particular entity; and
a second entity-specific feature score based on an analysis of the timing or sequencing of communications by the particular entity;
generate an entity profile associated with the particular entity, the entity profile including the plurality of entity-specific feature scores;
generate an anomaly score based on the entity profile; and
detect an anomaly if the anomaly score satisfies a specified criterion.
30. A non-transient computer readable medium containing instructions for causing a computer system to:
receive event data associated with a communication between an internal entity within a computer network and an external entity outside the computer network;
generate a plurality of entity-specific feature scores by processing the event data, wherein each of the entity-specific features scores is representative of a quantified evaluation of a level of risk associated with a particular entity, each of the feature scores generated by a different one of a plurality of different types of analyses of the event data, the particular entity being the internal entity or the external entity;
wherein the plurality of entity-specific feature scores include:
a first entity-specific feature score based on a lexical analysis of an identifier associated with the particular entity; and
a second entity-specific feature score based on an analysis of the timing or sequencing of communications by the particular entity;
generate an entity profile associated with the particular entity, the entity profile including the plurality of entity-specific feature scores;
generate an anomaly score based on the entity profile; and
detect an anomaly if the anomaly score satisfies a specified criterion.