Enterprise security graph
A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A method comprising:
receiving, by a computer system, event data representing a plurality of events on a computer network, the event data being indicative of a plurality of entities and at least one anomaly involved in the events;
acquiring, for each event, an event-specific relationship graph indicative of entities involved in the event and one or more relationships between the entities involved in the event, each event-specific relationship graph including a plurality of nodes and a plurality of edges interconnecting the nodes, the nodes representing the entities involved in the event, each edge representing an interaction between a pair of entities involved in the event;
acquiring anomaly data indicative of a plurality of security-related anomalies detected from the event data;
combining the event-specific relationship graphs for the plurality of events with the anomaly data into a composite relationship graph, the composite relationship graph including nodes that represent the entities involved in the plurality of events and nodes that represent the anomalies detected based on the event data, wherein the entities involved in the plurality of events include at least two types of entities, the composite relationship graph further including edges that represent the relationships between the entities involved in the plurality of events and the anomalies; and
detecting, by the computer system, a security threat by processing at least a portion of the composite relationship graph with a decision engine.
2. The method of claim 1 , wherein the event data are machine data.
3. The method of claim 1 , wherein the event data are timestamped machine data.
4. The method of claim 1 , wherein at least one of said receiving event data, creating a data structure, combining the relationship graphs and identifying a security threat is performed by processing logic of a machine learning model.
5. The method of claim 1 , wherein at least one of said receiving event data, said creating a data structure, combining the relationship graphs and identifying a security threat is performed in real time as the event data are received.
6. The method of claim 1 , wherein at least one of said receiving event data, creating a data structure, combining the relationship graphs and identifying a security threat is performed in a batch mode based on the event data retrieved from persistent storage.
7. The method of claim 1 , wherein the entities include at least one of a device in the computer network, a user of the device in the computer network, a website, and a data file stored in the device in the computer network.
8. The method of claim 1 , wherein the decision engine is a machine learning model.
9. The method of claim 1 , wherein said step of combining the relationship graphs comprises:
transferring the relationship graphs from the memory of the computer system to non-volatile storage at a predetermined periodicity.
10. The method of claim 1 , wherein said step of combining the relationship graphs comprises:
for each event, identifying one or more computer network activities of a particular type associated with an entity that occur during a time period, based on the data structure representing the relationship graph for the event;
combining the computer network activities of the particular type associated with the entity involved in the plurality of events into a combined computer network activity and counting the computer network activities that occur during a predefined time period; and
storing the combined computer network activity and the count into a data file designated for the particular type and the time period.
11. The method of claim 1 , wherein said step of combining the relationship graphs comprises:
for each event, identifying one or more computer network activities of a particular type associated with an entity that occur during a time period, based on the data structure representing the relationship graph for the event;
combining the computer network activities of the particular type associated with the entity involved in the plurality of events into a combined computer network activity and counting the computer network activities that occur during the time period;
storing the combined computer network activity and count; and
in response to detecting that combined computer network activities associated with the projection have timestamps that are older than a threshold time value, storing the combined network activities in a designated data container.
12. The method of claim 1 , wherein said step of combining the relationship graphs comprises:
for each event, identifying one or more computer network activities of a particular type associated with an entity that occur during a time period, based on the data structure representing the relationship graph for the event; and
combining the computer network activities of the particular type associated with the entity involved in the plurality of events into a combined computer network activity and recording the combined computer network activity, the combined computer network activity including an activity type, an originating entity, a target entity, the number of times the computer network activities occur in the time period, a start time, an end time, an average gap period between the computer network activities that occur in the time period, or a standard deviation of gap periods between the computer network activities that occur in the time period.
13. The method of claim 1 , wherein said step of combining the relationship graphs comprises:
for each event, identifying one or more computer network activities of a particular type associated with an entity that occur during a time period based on the data structure representing the relationship graph for the event;
combining the computer network activities of the particular type associated with the entity involved in the plurality of event into a combined computer network activity and recording the number of computer network activities that occur during the time period; and
determining, by the computer system, an association between the combined computer network activity of the particular type and a particular projection of the composite relationship graph, the composite relationship graph including a plurality of projections that correspond to a plurality of categories of computer network activities; and
storing the combined computer network activity into a designated data container.
14. The method of claim 1 , wherein the composite relationship graph includes a plurality of projections, one of the projections being a subset of the composite relationship graph that includes edges representing a plurality of anomaly activities conducted by users.
15. The method of claim 1 , wherein the composite relationship graph includes a plurality of projections, each of the projections being a subset of the composite relationship graph that includes edges representing a plurality of computer network activities of a particular category, the particular category of network activities corresponding to at least one of users logging into devices in the computer network, users visiting websites, users accessing files stored devices in the computer network, and users conducting anomaly activities.
16. The method of claim 1 , further comprising:
in response to recognizing that an anomaly has been detected from the event data, storing event data associated with the anomaly in non-volatile storage; and
generating a visualization of the event data associated with the anomaly in response to an instruction to visualize details of the anomaly.
17. The method of claim 1 , wherein the composite relationship graph includes a plurality of projections, each of the projection being a subset of the composite relationship graph that includes edges representing a plurality of computer network activities of a particular category, the projection being stored in a cluster of storage devices based on timestamps of combined computer network activities stored in data containers.
18. The method of claim 1 , further comprising:
identifying computer network activities represented in a projection of the composite relationship graph, the computer network activities having timestamps that satisfy a specified closeness criterion, wherein the composite relationship graph includes multiple projections, each of the projections being a subset of the composite relationship graph that includes edges representing a plurality of computer network activities of a particular category; and
storing the identified computer network activities in proximity to each other in non-volatile storage.
19. The method of claim 1 , further comprising:
receiving, at a predetermined periodicity, from one or more machine learning models that identify security threats, requests for data from a projection of the composite relationship graph and corresponding to a time range, wherein the composite relationship graph includes multiple projections that correspond to multiple categories of computer network activities.
20. The method of claim 1 , further comprising:
receiving, at a predetermined periodicity from one or more machine learning models that identify security threats, requests for combined computer network activities from a projection of the composite relationship graph and corresponding to a time range, wherein the composite relationship graph includes multiple projections that correspond to multiple categories of computer network activities, and the combined computer network activities include information about computer network activities from multiple events.
21. The method of claim 1 , further comprising:
receiving, at a predetermined periodicity from multiple machine learning models that identify security threats, requests for data from a common range of projections of the composite relationship graph and corresponding to a time range, wherein the composite relationship graph includes multiple projections that correspond to multiple categories of computer network activities; and
combining the data within the common range of projections and the common range of timestamps into a single input data structure as an input for the multiple machine learning models.
22. The method of claim 1 , wherein one or more machine learning models receive at least portions of the composite relationship graph as inputs and identify security threats and anomalies based thereon, and wherein the one or more machine learning models are executed in a batch mode.
23. The method of claim 1 , wherein one or more machine learning models receive at least portions of the composite relationship graph as inputs and identify security threats and anomalies based thereon, and wherein the one or more machine learning models are executed in a batch mode; and
the method further comprising:
storing the anomalies identified by the one or more machine learning models with an anomaly projection of the composite relationship graph; and
storing the event data associated the identified anomalies in non-volatile storage.
24. The method of claim 1 , wherein said identifying a security threat comprises:
converting at least a portion of the composite relationship graph into an anomaly relationship graph, the anomaly relationship graph including anomaly nodes that represent anomalies and entity nodes that represent entities in the computer network;
inputting the anomaly relationship graph into the decision engine, the decision engine having a machine learning model;
identifying, by the machine learning model, a set of anomaly nodes interconnecting entities nodes that form a neighborhood cluster in the anomaly data structure; and
identifying, by the machine learning model, a set of anomalies represented by the identified anomaly nodes as the security threat.
25. The method of claim 1 , wherein said identifying a security threat comprises:
converting at least a portion of the composite relationship graph into an anomaly relationship graph, the anomaly relationship graph including anomaly nodes that represent anomalies and entity nodes that represent entities in the computer network;
inputting the anomaly relationship graph into the decision engine, the decision engine being a machine learning model;
identifying, by the machine learning model, a subset of the anomaly relationship graph including anomaly nodes and entity nodes that have no relationships with other nodes outside of the subset in the anomaly relationship graph; and
identifying, by the machine learning model, a set of anomalies represented by anomaly nodes within the subset as the security threat.
26. The method of claim 1 , wherein said identifying a security threat comprises:
converting at least a portion of the composite relationship graph into an anomaly relationship graph, the anomaly relationship graph including anomaly nodes that represent anomalies and entity nodes that represent entities in the computer network;
inputting the anomaly relationship graph into the decision engine, the decision engine being a machine learning model;
identifying, by the machine learning model, a set of anomaly nodes directly connecting a particular entity node in the anomaly relationship graph; and
identifying, by the machine learning model, a set of anomalies represented by the identified anomaly nodes as the security threat.
27. The method of claim 1 , wherein said identifying a security threat comprises:
converting at least a portion of the composite relationship graph into an anomaly relationship graph, the anomaly relationship graph including anomaly nodes that represent anomalies and entity nodes that represent entities in the computer network;
inputting the anomaly relationship graph into the decision engine, the decision engine being a machine learning model;
identifying, by the machine learning model, a set of anomaly nodes directly connecting a particular entity node in the anomaly relationship graph;
further identify a group of anomaly nodes from the identified plurality of anomaly nodes, the group of anomaly nodes having timestamps that satisfy a specific closeness criterion; and
identifying, by the machine learning model, a set of anomalies represented by the identified group of anomaly nodes as the security threat.
28. The method of claim 1 , wherein said identifying a security threat comprises:
identifying, by the decision engine, a set of anomalies as the security threat based on the composite relationship graph;
assigning the anomalies into categories of internal anomaly, malware anomaly, incoming anomaly and exfiltration anomaly; and
confirming that the anomaly network activities form a security threat by applying a security rule to the anomalies based on the assigned categories of the anomalies.
29. A computing device comprising:
a processor; and
a memory storing instructions that, when executed by the processor, cause the computing device to perform a process including:
receiving event data representing each of a plurality of events on a computer network, the event data being indicative of a plurality of entities and at least one anomaly involved in the events;
acquiring, for each event, an event-specific relationship graph indicative of entities involved in the event and at least one relationship between the entities involved in the event, each event-specific relationship graph including a plurality of nodes and a plurality of edges interconnecting the nodes, the nodes representing the entities involved in the event, each edge representing an interaction between a pair of entities involved in the event;
acquiring anomaly data indicative of a plurality of security-related anomalies detected from the event data;
combining the event-specific relationship graphs for the plurality of events with the anomaly data into a composite relationship graph, the composite relationship graph including nodes that represent the entities involved in the plurality of events and nodes that represent the anomalies detected based on the event data, wherein the entities involved in the plurality of events include at least two types of entities, the composite relationship graph further including edges that represent the relationships between the entities involved in the plurality of events and the anomalies; and
detecting a security threat by processing at least a portion of the composite relationship graph with a decision engine.
30. A non-transitory machine readable storage medium storing instructions, execution of which in a machine causes the machine to perform a process including:
receiving event data representing each of a plurality of events on a computer network, the event data being indicative of a plurality of entities and at least one anomaly involved in the events;
acquiring, for each event, an event-specific relationship graph indicative of entities involved in the event and at least one relationship between the entities involved in the event, each event-specific relationship graph including a plurality of nodes and a plurality of edges interconnecting the nodes, the nodes representing the entities involved in the event, each edge representing an interaction between a pair of entities involved in the event;
acquiring anomaly data indicative of a plurality of security-related anomalies detected from the event data;
combining the event-specific relationship graphs for the plurality of events with the anomaly data into a composite relationship graph, the composite relationship graph including nodes that represent the entities involved in the plurality of events and nodes that represent the anomalies detected based on the event data, wherein the entities involved in the plurality of events include at least two types of entities, the composite relationship graph further including edges that represent the relationships between the entities involved in the plurality of events and the anomalies; and
detecting a security threat by processing at least a portion of the composite relationship graph with a decision engine.