Composite relationship graph for network security
A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.
1. A method comprising:
receiving, by a computer system, event data representing a plurality of events on a computer network, the event data corresponding to a plurality of entities and at least one anomaly corresponding to the events;
acquiring, for each event, an event-specific relationship graph indicative of entities involved in the event and one or more relationships between the entities involved in the event, each event-specific relationship graph including a plurality of nodes and one or more edges interconnecting nodes of the plurality of nodes, the nodes representing the entities involved in the event, each edge representing an interaction between a pair of entities involved in the event;
acquiring anomaly data indicative of a plurality of security-related anomalies;
using the event-specific relationship graphs for the plurality of events and the anomaly data to form a composite relationship graph, the composite relationship graph including nodes that represent the entities involved in the plurality of events and nodes that represent the anomalies, the composite relationship graph further including edges that represent the relationships between the entities involved in the plurality of events and the anomalies; and
detecting, by the computer system, a security threat by using the composite relationship graph.
2. The method of claim 1 , wherein the event data comprise timestamped machine data.
3. The method of claim 1 , wherein at least a portion of said method is performed by processing logic of a machine learning model.
4. The method of claim 1 , wherein the plurality of entities includes a device in the computer network, a user of the device in the computer network, and a data file stored in the device in the computer network.
5. The method of claim 1 , wherein said using the event-specific relationship graphs for the plurality of events and the anomaly data to form a composite relationship graph comprises:
for each event of the plurality of events, identifying one or more computer network activities of a particular type associated with an entity that occur during a time period, based on the relationship graph for the event;
combining the computer network activities of the particular type associated with the entity involved in the plurality of events into a combined computer network activity and counting the computer network activities that occur during a predefined time period; and
storing the combined computer network activity and the count into a data file designated for the particular type and the time period.
6. The method of claim 1 , wherein said using the event-specific relationship graphs for the plurality of events and the anomaly data to form a composite relationship graph comprises:
for each event, identifying one or more computer network activities of a particular type associated with an entity that occur during a time period, based on a data structure representing the relationship graph for the event;
combining the computer network activities of the particular type associated with the entity involved in the plurality of events into a combined computer network activity and determining a count of the computer network activities that occur during the time period;
storing the combined computer network activity and the count; and
in response to detecting that combined computer network activities associated with the projection have timestamps that are older than a threshold time value, storing a combined network activities projection in a designated data container.
7. The method of claim 1 , wherein said using the event-specific relationship graphs for the plurality of events and the anomaly data to form a composite relationship graph comprises:
for each event, identifying one or more computer network activities of a particular type associated with an entity that occur during a time period, based on a data structure representing the relationship graph for the event; and
combining the computer network activities of the particular type associated with the entity involved in the plurality of events into a combined computer network activity and recording the combined computer network activity, the combined computer network activity including an activity type, an originating entity, a target entity, a number of times the computer network activities occur in the time period, a start time, an end time, and an average gap period between the computer network activities that occur in the time period or a standard deviation of gap periods between the computer network activities that occur in the time period.
8. The method of claim 1 , wherein said using the event-specific relationship graphs for the plurality of events and the anomaly data to form a composite relationship graph comprises:
for each event, identifying one or more computer network activities of a particular type associated with an entity that occur during a time period based on a data structure representing the relationship graph for the event;
combining the computer network activities of the particular type associated with the entity involved in the plurality of event into a combined computer network activity and recording a number of computer network activities that occur during the time period; and
determining, by the computer system, an association between the combined computer network activity of the particular type and a particular projection of the composite relationship graph, the composite relationship graph including a plurality of projections that correspond to a plurality of categories of computer network activities; and
storing the combined computer network activity into a designated data container.
9. The method of claim 1 , wherein the composite relationship graph includes a plurality of projections, one of the projections being a subset of the composite relationship graph that includes edges representing a plurality of anomaly activities conducted by users.
10. The method of claim 1 , further comprising:
in response to recognizing that an anomaly has been detected from the event data, storing event data associated with the anomaly in non-volatile storage; and
generating a visualization of the event data associated with the anomaly in response to an instruction to visualize details of the anomaly.
11. The method of claim 1 , further comprising:
receiving, at a predetermined periodicity from one or more machine learning models that identify security threats, requests for combined computer network activities from a projection of the composite relationship graph and corresponding to a time range, wherein the composite relationship graph includes multiple projections that correspond to multiple categories of computer network activities, and the combined computer network activities include information about computer network activities from multiple events.
12. The method of claim 1 , further comprising:
receiving, at a predetermined periodicity from a plurality of machine learning models that identify security threats, requests for data from a common range of projections of the composite relationship graph and corresponding to a time range, wherein the composite relationship graph includes multiple projections that correspond to multiple categories of computer network activities; and
combining the data within the common range of projections and a common range of timestamps into a single input data structure as an input for the plurality of machine learning models.
13. The method of claim 1 , wherein a machine learning model receives at least a portion of the composite relationship graph as input and identifies the security threat based on the composite relationship graph.
14. The method of claim 1 , wherein identifying the security threat comprises:
converting at least a portion of the composite relationship graph into an anomaly relationship graph, the anomaly relationship graph including anomaly nodes that represent anomalies and entity nodes that represent entities in the computer network;
inputting the anomaly relationship graph into a decision engine, the decision engine having a machine learning model;
identifying, by the machine learning model, a set of anomaly nodes interconnecting entities nodes that form a neighborhood cluster in an anomaly data structure; and
identifying, by the machine learning model, a set of anomalies represented by the identified anomaly nodes as the security threat.
15. The method of claim 1 , wherein identifying the security threat comprises:
converting at least a portion of the composite relationship graph into an anomaly relationship graph, the anomaly relationship graph including anomaly nodes that represent anomalies and entity nodes that represent entities in the computer network;
inputting the anomaly relationship graph into a decision engine, the decision engine being a machine learning model;
identifying, by the machine learning model, a set of anomaly nodes directly connecting a particular entity node in the anomaly relationship graph;
identifying a group of anomaly nodes from the identified set of anomaly nodes; and
identifying, by the machine learning model, a set of anomalies represented by the identified group of anomaly nodes as the security threat.
16. The method of claim 1 , wherein identifying the security threat comprises:
converting at least a portion of the composite relationship graph into an anomaly relationship graph, the anomaly relationship graph including anomaly nodes that represent anomalies and entity nodes that represent entities in the computer network;
inputting the anomaly relationship graph into a decision engine, the decision engine being a machine learning model;
identifying, by the machine learning model, a set of anomaly nodes directly connecting a particular entity node in the anomaly relationship graph;
identifying a group of anomaly nodes from the identified set of anomaly nodes, the group of anomaly nodes having timestamps that satisfy a specific closeness criterion; and
identifying, by the machine learning model, a set of anomalies represented by the identified group of anomaly nodes as the security threat.
17. The method of claim 1 , wherein identifying the security threat comprises:
identifying, by the decision engine, a set of anomalies as the security threat based on the composite relationship graph;
assigning the anomalies into categories of internal anomaly, malware anomaly, incoming anomaly and exfiltration anomaly; and
confirming that the anomalies form a security threat by applying a security rule to the anomalies based on the assigned categories of the anomalies.
18. The method of claim 1 , wherein the entities involved in the plurality of events include at least two types of entities.
19. A computing device comprising:
a processor; and
a memory storing instructions, execution of which by the processor causes the computing device to perform a process including:
receiving, by a computer system, event data representing a plurality of events on a computer network, the event data corresponding to a plurality of entities and at least one anomaly corresponding to the events;
acquiring, for each event, an event-specific relationship graph indicative of entities involved in the event and one or more relationships between the entities involved in the event, each event-specific relationship graph including a plurality of nodes and one or more edges interconnecting nodes of the plurality of nodes, the nodes representing the entities involved in the event, each edge representing an interaction between a pair of entities involved in the event;
acquiring anomaly data indicative of a plurality of security-related anomalies;
using the event-specific relationship graphs for the plurality of events and the anomaly data to form a composite relationship graph, the composite relationship graph including nodes that represent the entities involved in the plurality of events and nodes that represent the anomalies, the composite relationship graph further including edges that represent the relationships between the entities involved in the plurality of events and the anomalies; and
detecting, by the computer system, a security threat by using the composite relationship graph.
20. A non-transitory machine readable storage medium storing instructions, execution of which in a machine causes the machine to perform a process including:
receiving, by a computer system, event data representing a plurality of events on a computer network, the event data corresponding to a plurality of entities and at least one anomaly corresponding to the events;
acquiring, for each event, an event-specific relationship graph indicative of entities involved in the event and one or more relationships between the entities involved in the event, each event-specific relationship graph including a plurality of nodes and one or more edges interconnecting nodes of the plurality of nodes, the nodes representing the entities involved in the event, each edge representing an interaction between a pair of entities involved in the event;
acquiring anomaly data indicative of a plurality of security-related anomalies;
using the event-specific relationship graphs for the plurality of events and the anomaly data to form a composite relationship graph, the composite relationship graph including nodes that represent the entities involved in the plurality of events and nodes that represent the anomalies, the composite relationship graph further including edges that represent the relationships between the entities involved in the plurality of events and the anomalies; and
detecting, by the computer system, a security threat by using the composite relationship graph.