System and method for identification of unknown illicit networks
A method and system for identifying unknown illicit networks uses node data, link data, and network data in a recursive analysis that computes node probabilities by combining patterns based on nodes, links and the topology of the network structure present simultaneously in the data. An iterative balancing algorithm is used to make the probability values self-consistent.
1. A computer implemented method for identifying unknown illicit networks, a computer performing steps comprising:
obtaining by the computer node data and link data for a plurality of nodes, the node data comprising attributes of a node, there being one or more links, each link representing an interaction between nodes and the link data comprising attributes associated with the link;
training by the computer a node-based probability model iteratively using the node data and the link data, iterating until a stop condition is satisfied, each iteration recursively using a probability of each node as input for computation of a probability of each other node;
training by the computer a topology-based model using network data; and
solving by the computer a maximum likelihood problem based on said node probabilities from said recursive iteration, using as constraints output of said topology-based model, and determining whether each of said nodes is illicit (bad') or not ('good').
2. The method of claim 1 , wherein said node-based probability model is a conditional probability based model.
3. The method of claim 1 , wherein said node-based probability model is a cluster-based model.
4. The method of claim 1 , wherein the step of solving a maximum likelihood problem employs an expert to evaluate what-if scenarios using different values of the constraints.
5. The method of claim 3 , wherein an algorithm for clustering assumes that a node's probability depends only on a total number of ‘good’/‘bad’ connections to each cluster and uses as a classifier a logistic regression of:
p
ij
=
exp
(
a
0
+
∑
m
∈
L
,
s
=
s
(
m
)
(
a
s
+
p
m
+
a
s
-
(
1
-
p
m
)
)
)
1
+
exp
(
a
0
+
∑
m
∈
L
,
s
=
s
(
m
)
(
a
s
+
p
m
+
a
s
-
(
1
-
p
m
)
)
)
where p ij is the probability for the i th node on the j th network, p m is the probability for the m th link, and a s are the coefficients of probability for the s th node.
6. The method of claim 2 , wherein said conditional probability based model has two components, a first component being a conditional probability that a node is ‘bad’ under a condition that the node has a neighbor (NB) connected via a link L, and a second component being a probability based only on attributes of the node.
7. The method of claim 1 , wherein historical datasets are used to provide additional node data and additional link data.
8. The method of claim 1 , wherein historical data corresponding to a node's close neighborhood is available and corresponding Eigen values are used as predictive attributes in the topology-based model.
9. A computer implemented system for identifying unknown illicit networks, comprising:
a data processing tool for obtaining from raw data node data and link data for a plurality of nodes, the node data comprising attributes of a node, there being one or more links, each link representing an interaction between nodes and the link data comprising attributes associated with the link;
a first analytic subsystem for training a node-based probability model iteratively using the node data and the link data, iterating until a stop condition is satisfied, each iteration recursively using a probability of each node as input for computation of a probability of each other node;
a second analytic subsystem for training a topology-based model using network data; and
a third analytic subsystem for solving a maximum likelihood problem based on said node probabilities from said recursive iteration, using as constraints output of said topology-based model, and determining whether each of said nodes is illicit (‘bad’) or not (‘good’).
10. The system of claim 9 , wherein said node-based probability model is a conditional probability based model.
11. The system of claim 9 , wherein said node-based probability model is a cluster-based model.
12. The system of claim 9 , wherein the third analytic subsystem for solving a maximum likelihood problem is adapted to employ an expert to evaluate what-if scenarios using different values of the constraints.
13. The system of claim 11 , wherein an algorithm for clustering assumes that a node's probability depends only on a total number of ‘good’/‘bad’ connections to each cluster and uses as a classifier a logistic regression of:
p
ij
=
exp
(
a
0
+
∑
m
∈
L
,
s
=
s
(
m
)
(
a
s
+
p
m
+
a
s
-
(
1
-
p
m
)
)
)
1
+
exp
(
a
0
+
∑
m
∈
L
,
s
=
s
(
m
)
(
a
s
+
p
m
+
a
s
-
(
1
-
p
m
)
)
)
where p ij is the probability for the i th node on the j th network, p m is the probability for the m th link, and a s are the coefficients of probability for the s th node.
14. The system of claim 10 , wherein said conditional probability based model has two components, a first component being a conditional probability that a node is ‘bad’ under a condition that the node has a neighbor (NB) connected via a link L, and a second component being a probability based only on attributes of the node.
15. The system of claim 9 , wherein historical datasets are used to provide additional node data and additional link data.
16. The system of claim 9 , wherein historical data corresponding to a node's close neighborhood is available and corresponding Eigen values are used as predictive attributes in the topology-based model.
17. A server having software modules stored thereon for identifying unknown illicit networks, and having a processor operable to execute the software modules, the software modules comprising:
first computer code for obtaining node data and link data for a plurality of nodes, the node data comprising attributes of a node, there being one or more links, each link representing an interaction between nodes and the link data comprising attributes associated with the link;
second computer code for training a node-based probability model iteratively using the node data and the link data, iterating until a stop condition is satisfied, each iteration recursively using a probability of each node as input for computation of a probability of each other node;
third computer code for training a topology-based model using network data; and
fourth computer code for solving a maximum likelihood problem based on said node probabilities from said recursive iteration, using as constraints output of said topology-based model, and determining whether each of said nodes is illicit (‘bad’) or not (‘good’).
18. A server having software modules stored thereon for identifying unknown illicit networks as in claim 17 , wherein said node-based probability model is a conditional probability based model.
19. A server having software modules stored thereon for identifying unknown illicit networks as in claim 17 , wherein said node-based probability model is a cluster-based model.
20. A server having software modules stored thereon for identifying unknown illicit networks as in claim 17 , wherein fourth computer code for solving a maximum likelihood problem is adapted to employ an expert to evaluate what-if scenarios using different values of the constraints.