Graph-based network security threat detection across time and entities
The disclosed techniques relate to a graph-based network security analytic framework to combine multiple sources of information and security knowledge in order to detect risky behaviors and potential threats. In some examples, the input can be anomaly events or simply regular events. The entities associated with the activities can be grouped into smaller time units, e.g., per day. The riskiest days of activity can be found by computing a risk score for each day and according to the features in the day. A graph can be built with links between the time units. The links can also receive scoring based on a number of factors. The resulting graph can be compared with known security knowledge for adjustments. Threats can be detected based on the adjusted risk score for a component (i.e., a group of linked entities) as well as a number of other factors.
1. A method comprising:
accessing, from a data store, a relationship graph corresponding to a time range, the time range having a number of time units, the relationship graph having entities as nodes and relationships among the entities as links, the relationship graph reflecting a batch of events that occurred during the time range, wherein each event of the batch of events includes timestamped, raw machine data that reflects one or more of: (1) activity occurred in an information technology (IT) or a security technology environment, (2) a time at which the activity occurred, and (3) a number of entities associated with the activity;
assigning the nodes in the relationship graph to groups based on event timestamps, each group corresponding to a time unit and including nodes associated with activities that occurred in the time unit;
constructing links for nodes between different groups, each link representing a relationship between nodes as established by a respective activity recorded in the batch of events, each chain of linked nodes forming a component;
computing a total interest score for each of the formed components, wherein the total interest score reflects a totality of interest generated from all nodes attached to a given link;
adjusting the total interest score for each of the formed components based on comparing events underlying a component with a pattern of interest, wherein the pattern of interest identifies an expected temporal order and/or logical relationship in underlying events for such component to be of interest; and
identifying a component for further security scrutiny based on the adjusted total interest score.
2. The method of claim 1 , wherein the plurality of events comprise events that have been earmarked as anomalies.
3. The method of claim 1 , wherein each node carries an anomaly score that is assigned from a previous data analytic stage.
4. The method of claim 1 , wherein the relationship graph is a subset of a composite relationship graph that includes edges representing a plurality of anomaly activities conducted by entities.
5. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including:
generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group.
6. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including:
generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group,
wherein a feature in the set of features carries a different weight than another feature.
7. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including:
generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group,
wherein the total interest score for a formed component factors in the group interest score of the groups to which the nodes in the formed component belong.
8. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including:
generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group; and
ranking the number of groups based on their group interest scores, wherein only a predetermined number of top ranked groups are further processed for constructing links for nodes between different groups.
9. The method of claim 1 , further comprising: determining a group interest score for each of the groups, based on steps including:
generating the group interest score based on a set of features from a respective group, wherein the set of features are identified by a predetermined list of features that are characteristic of the activities recorded in the events in the respective group;
ranking the number of groups based on their group interest scores, wherein only a predetermined number of top ranked groups are further processed for constructing links for nodes between different groups; and
performing clustering for the number of groups after normalizing values in the set of features in each group.
10. The method of claim 1 , further comprising:
determining a link score for each link in the formed components.
11. The method of claim 1 , further comprising:
determining a link score for each link in the formed components,
wherein the link score is determined based on a number of common nodes between the groups with which the formed component is associated.
12. The method of claim 1 , further comprising:
determining a link score for each link in the formed components,
wherein the link score is determined based on a distance in time between the groups with which the formed component is associated.
13. The method of claim 1 , further comprising:
determining a link score for each link in the formed components,
wherein the link score is determined based on an anomaly score of each node in the formed component.
14. The method of claim 1 , further comprising:
determining a link score for each link in the formed components,
wherein the total interest score for the formed component factors in the link score of the link that connects the nodes in the formed component.
15. The method of claim 1 , further comprising:
creating a new graph using the formed components, wherein the new graph includes the nodes with respective links and corresponding groups.
16. The method of claim 1 , further comprising:
creating a new graph using the formed components, wherein the new graph includes the nodes with respective links and corresponding group,
wherein the nodes in the new graph are coupled to underlying events so that, responsive to a request, the underlying events are produced as supporting evidence.
17. The method of claim 1 , further comprising:
before assigning nodes to groups, filtering the nodes and links in the relationship graph by removing nodes that include a whitelisted entity.
18. The method of claim 1 , further comprising:
before assigning nodes to groups, filtering the nodes and links in the relationship graph by removing nodes that include an entity having an exceeding number of anomaly links to other entities as compared to a threshold.
19. The method of claim 1 , wherein the total interest score for a formed component increases exponentially when the events underlying the formed component matches the pattern of interest.
20. The method of claim 1 , wherein the total interest score for a formed component increases exponentially when the events underlying the formed component matches the pattern of interest,
wherein the total interest score for a formed component decreases exponentially when the events underlying the formed component mismatches the pattern of interest.
21. The method of claim 1 , wherein the pattern of interest includes definitions for a sequence and an anti-sequence associated with an anomaly.
22. The method of claim 1 , wherein the pattern of interest includes definitions for a sequence and an anti-sequence associated with an anomaly,
wherein the total interest score for a formed component decreases exponentially when the events underlying the formed component matches the anti-sequence.
23. The method of claim 1 , wherein the pattern of interest includes a malware installation followed by a file transfer or a beaconing anomaly.
24. The method of claim 1 , further comprising: performing a network security related action on the identified component.
25. The method of claim 1 , wherein the entities are users, computing devices, or any combination thereof.
26. The method of claim 1 , wherein the time range is more than one day, and wherein the time unit is one day.
27. The method of claim 1 , wherein steps recited in the method are repeated at a predetermined periodicity.
28. The method of claim 1 , wherein steps recited in the method are performed by a batch analysis engine that is implemented using APACHE SPARK™, and wherein the data store is implemented using APACHE HADOOP™.
29. A computer system comprising:
a processor; and
a communication device, operatively coupled to the processor, through which to receive first event data indicative of computer network activity of an entity that is part of or interacts with a computer network and second event data indicative of additional computer network activity associated with the entity;
wherein the processor is configured to perform steps including:
accessing, from a data store, a relationship graph corresponding to a time range, the time range having a number of time units, the relationship graph having entities as nodes and relationships among the entities as links, the relationship graph reflecting a batch of events that occurred during the time range, wherein each event of the batch of events includes timestamped, raw machine data that reflects one or more of: (1) activity occurred in an information technology (IT) or a security technology environment, (2) a time at which the activity occurred, and (3) a number of entities associated with the activity;
assigning the nodes in the relationship graph to groups based on event timestamps, each group corresponds to a time unit and including nodes associated with activities that occurred in the time unit;
constructing links for nodes between different groups, each link representing a relationship between nodes as established by a respective activity recorded in the batch of events, each chain of linked nodes forming a component;
computing a total interest score for each of formed components, wherein the total interest score reflects a totality of interest generated from all nodes attached to a given link;
adjusting the total interest score for each of the formed components based on comparing events underlying a component with a pattern of interest, wherein the pattern of interest identifies an expected temporal order and/or logical relationship in underlying events for such component to be of interest;
identifying a component for further security scrutiny based on the adjusted total interest score.
30. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:
accessing, from a data store, a relationship graph corresponding to a time range, the time range having a number of time units, the relationship graph having entities as nodes and relationships among the entities as links, the relationship graph reflecting a batch of events that occurred during the time range, wherein each event of the batch of events includes timestamped, raw machine data that reflects one or more of: (1) activity occurred in an information technology (IT) or a security technology environment, (2) a time at which the activity occurred, and (3) a number of entities associated with the activity;
assigning the nodes in the relationship graph to groups based on event timestamps, each group corresponds to a time unit and including nodes associated with activities that occurred in the time unit;
constructing links for nodes between different groups, each link representing a relationship between nodes as established by a respective activity recorded in the batch of events, each chain of linked nodes forming a component;
computing a total interest score for each of formed components, wherein the total interest score reflects a totality of interest generated from all nodes attached to a given link;
adjusting the total interest score for each of the formed components based on comparing events underlying a component with a pattern of interest, wherein the pattern of interest identifies an expected temporal order and/or logical relationship in underlying events for such component to be of interest;
identifying a component for further security scrutiny based on the adjusted total interest score.