Detection of network anomalies based on relationship graphs
The disclosed techniques relate to a graph-based network security analytic framework to combine multiple sources of information and security knowledge in order to detect risky behaviors and potential threats. In some examples, the input can be anomaly events or simply regular events. The entities associated with the activities can be grouped into smaller time units, e.g., per day. The riskiest days of activity can be found by computing a risk score for each day and according to the features in the day. A graph can be built with links between the time units. The links can also receive scoring based on a number of factors. The resulting graph can be compared with known security knowledge for adjustments. Threats can be detected based on the adjusted risk score for a component (i.e., a group of linked entities) as well as a number of other factors.
1. A method comprising:
accessing a relationship graph in which entities associated with an information technology network are represented as nodes and relationships among the nodes are represented as links;
assigning the nodes in the relationship graph to groups to form a plurality of groups, each group of the plurality of groups including nodes associated with activities that share a particular characteristic;
for each of the groups, determining a group interest score for the group based on a set of features from the group, wherein the set of features are identified based on a specified list of features that are characteristic of the activities recorded in events in the respective group;
constructing links between nodes across different groups of the plurality of groups, to form a chain of linked nodes, the chain of linked nodes forming a component;
computing a score for the component, wherein the score is indicative of a level of interest associated with nodes attached to a given link; and
identifying the component for security scrutiny based on the score for the component.
2. The method of claim 1 , further comprising:
adjusting the score for the component based on comparing events underlying the component with a pattern of interest, wherein the pattern of interest identifies an expected temporal order and/or logical relationship in underlying events for the component to be of interest.
3. The method of claim 1 , wherein the plurality of events comprise events that have been earmarked as anomalies, and wherein each node is assigned an anomaly score from a previous data analytic stage.
4. The method of claim 1 , wherein the relationship graph is a subset of a composite relationship graph that includes edges representing a plurality of anomalous activities conducted by entities.
5. The method of claim 1 , wherein a feature in the set of features carries a different weight than another feature.
6. The method of claim 1 , wherein the score for the component factors in the group interest score of the groups to which the nodes in the component belong.
7. The method of claim 1 , further comprising:
ranking the plurality of groups based on their group interest scores, wherein only a specified number of top ranked groups are further processed for constructing links for nodes between different groups.
8. The method of claim 1 , further comprising:
ranking the plurality of groups based on their group interest scores, wherein only a specified number of top ranked groups are further processed for constructing links for nodes between different groups; and
performing clustering for the plurality of groups after normalizing values in the set of features in each group.
9. The method of claim 1 , further comprising:
determining a link score for the component.
10. The method of claim 1 , further comprising:
determining a link score for each link in the component, based on a number of common nodes between the groups with which the component is associated.
11. The method of claim 1 , further comprising:
determining a link score for the component, based on a distance in time between the groups with which the component is associated.
12. The method of claim 1 , further comprising:
determining a link score for the component, based on an anomaly score of each node in the component.
13. The method of claim 1 , further comprising:
determining a link score for each link in the component, wherein the score for the component factors in the link score of the link that connects the nodes in the component.
14. The method of claim 1 , further comprising: creating a new graph using the component.
15. The method of claim 1 , further comprising:
creating a new graph using the component, wherein the new graph includes nodes with respective links and a corresponding group, and wherein the nodes in the new graph are coupled to underlying events so that, responsive to a request, the underlying events are output as supporting evidence.
16. The method of claim 1 , further comprising:
before assigning the nodes in the relationship graph to groups, filtering the nodes and links in the relationship graph by removing nodes that include a whitelisted entity.
17. The method of claim 1 , further comprising:
before assigning the nodes in the relationship graph to groups, filtering the nodes and links in the relationship graph by removing nodes that include an entity having more than a threshold number of anomaly links to other entities.
18. The method of claim 1 , wherein the score for the component increases exponentially based on events underlying the component matching a pattern of interest.
19. The method of claim 1 , wherein the score for the component increases exponentially based on events underlying the component matching a pattern of interest, and
wherein the score for the component decreases exponentially based on events underlying the component mismatching the pattern of interest.
20. The method of claim 1 , further comprising: adjusting the score for the component based on comparing events underlying the component with a pattern of interest, wherein the pattern of interest includes definitions for a sequence and an anti-sequence associated with an anomaly.
21. The method of claim 1 , further comprising: adjusting the score for the component based on comparing events underlying the component with a pattern of interest, wherein the pattern of interest includes definitions for a sequence and an anti-sequence associated with an anomaly, and wherein the score for the component decreases exponentially when the events underlying the component matches the anti-sequence.
22. The method of claim 1 , further comprising: adjusting the score for the component based on comparing events underlying the component with a pattern of interest, wherein the pattern of interest includes a malware installation followed by a file transfer or a beaconing anomaly.
23. The method of claim 1 , further comprising: performing a network security related action on the component.
24. The method of claim 1 , wherein the entities are users, devices, or a combination thereof.
25. The method of claim 1 , wherein the relationship graph reflects a set of events that occurred on the information technology network during a time range, and a group of the plurality of groups includes nodes associated with activities that occurred during the time range.
26. The method of claim 1 , wherein steps of the method are repeated at a predetermined periodicity.
27. The method of claim 1 , wherein steps recited in the method are performed by a batch analysis engine that is implemented using APACHE SPARK™, and wherein the relationship graph is stored in a data store that is implemented using APACHE HADOOP®.
28. A computer system comprising:
a processor; and
a communication device, operatively coupled to the processor, through which to receive event data indicative of activity of entities associated with an information technology network;
wherein the processor is configured to perform steps including:
accessing a relationship graph in which the entities associated with the information technology network are represented as nodes and relationships among the nodes are represented as links;
assigning the nodes in the relationship graph to groups to form a plurality of groups, each group of the plurality of groups including nodes associated with activities that share a particular characteristic;
for each of the groups, determining a group interest score for the group based on a set of features from the group, wherein the set of features are identified based on a specified list of features that are characteristic of the activities recorded in events in the respective group;
constructing links between nodes across different groups of the plurality of groups, to form a chain of linked nodes, the chain of linked nodes forming a component;
computing a score for the component, wherein the score is indicative of a level of interest associated with nodes attached to a given link; and
identifying the component for security scrutiny based on the score for the component.
29. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, execution of which in the processing system causes the processing system to perform operations comprising:
accessing a relationship graph in which entities associated with an information technology network are represented as nodes and relationships among the nodes are represented as links;
assigning the nodes in the relationship graph to groups to form a plurality of groups, each group of the plurality of groups including nodes associated with activities that share a particular characteristic;
for each of the groups, determining a group interest score for the group based on a set of features from the group, wherein the set of features are identified based on a specified list of features that are characteristic of the activities recorded in events in the respective group;
constructing links between nodes across different groups of the plurality of groups, to form a chain of linked nodes, the chain of linked nodes forming a component;
computing a score for the component, wherein the score is indicative of a level of interest associated with nodes attached to a given link; and
identifying the component for security scrutiny based on the score for the component.