Method for clustering nodes of a textual network taking into account textual content, computer-readable storage device and system implementing said method
View Patent ↗The invention relates to a method for clustering nodes of a network, the network comprising nodes associated with message edges of text data, the method comprising an initialization step of determination of a first initial clustering of the nodes, and a step of iterative inference of a generative model of text documents. Edges are modeled with a Stochastic Block Model (SBM) and the sets of documents between and within clusters are modeled according to a generative model of documents. The inference step comprises iteratively modelling the text documents and the underlying topics of their textual content, and updating the clustering as a function of the modelling, until a convergence criterion is fulfilled and an optimized clustering and corresponding optimized values of the parameters of the models are output.
1. A computer-implemented method for clustering nodes of a communications network, said communications network comprising a number M of nodes, nodes i,j being associated with edges A ij of an adjacency matrix A,
a message edge W ij of text data by edge A ij ,
the message edge W ij being a vector W ij =(W ij 1 , . . . , W ij d , . . . , W ij Dij ), wherein D ij is a number of text documents exchanged between the node i and the node j, where W ij d is at least one set of documents and W ij d includes text of a d th message exchanged between the node i and the node j,
the method comprising:
1) a first step of initially clustering all the number M of nodes into a predetermined number Q of clusters, for determining a first estimated clustering Y=(Y 1 , Y 2 . . . ,Y M ;
2) a second step of inference of a generative model,
wherein edges A ij are modeled with a Stochastic Block Model (SBM) and, conditionally to an estimated clustering Y, the set of documents W ij d between and within the clusters are modeled according to a generative model of documents,
the second step of inference comprising the following sub-steps:
a. for a cluster q, determining from the first estimated clustering Y, an estimated proportion ρ q of the cluster q, an estimated probability π qr that a first node included in the cluster q and a second node included in a cluster r are associated with text data in the communications network;
b. determining from the first estimated clustering Y, an estimated distribution R(Z,θ), and an estimated probability β, wherein:
Z ij dn is a topic vector indicating a first topic of an n-th word n of the at least one set of documents W ij d , wherein W ij dn corresponds to the n-th word of a d-th document;
θ qr is a vector of topic proportions, wherein θ qrk is an estimated probability of a topic k of text data associated between the cluster q and the cluster r,
β kv is a probability of each word v of a dictionary being used in the topic k;
the sub-step b comprising the following sub-steps:
b1. for each cluster q, aggregating the at least one set of documents W ij d associated with each of a pairs of nodes (i,j) that are both within the cluster q, so as to produce a first aggregate of text data associated with the cluster q,
and for each cluster q, aggregating the at least one set of documents W ij d associated with each of the pairs of nodes (i, j), in which the nodes i belongs to the cluster q and the node j belongs to the cluster r, so as to produce at least one second aggregate associated to a pair of clusters (q,r),
b2. processing the first and second aggregates to determine the topics k corresponding to text data of the first and second aggregates, with a predetermined number K of topics being determined, and
b3. processing the text data of the first and second aggregates to determine a distribution R(Z,θ) and a probability β;
b4. iteratively repeating sub-steps b1, b2 and b3 of the second step 2 );
c. calculating a new estimated clustering Y of the nodes in Q clusters, in accordance with calculated probabilities β, ρ, π and the estimated distribution R(Z,θ) of the sub-steps a and b of the second step 2),
by determining, for each node i of the communications network, which optimal cluster provides a maximum value of a statistical criterion when each node i is assigned to said cluster, and assigning at least one node i to the determined optimal cluster;
3) a third step in which the second step 2) is repeated, with the new estimated clustering Y calculated in sub-step c and injected in the step 2), until a convergence criterion is fulfilled for determining and outputting a final clustering Y, at least one final proportions ρ q , a final distribution R(Z,θ), at least one final probability π qr , and at least one final probability β kv .
2. The method of claim 1 , wherein in the generative model of documents, the modeling of the edge A ij leads to the following joint distribution:
p
(
A
,
Y
❘
ρ
,
π
)
=
p
(
A
❘
Y
,
π
)
p
(
Y
❘
ρ
)
,
where
p
(
A
❘
Y
,
π
)
=
∏
i
≠
j
M
p
(
A
ij
❘
Y
i
,
Y
j
,
π
)
,
=
∏
i
≠
j
M
∏
q
,
r
Q
p
(
A
ij
❘
π
qr
)
Y
iq
Y
jr
,
and
p
(
Y
❘
ρ
)
=
∏
i
=
1
M
p
(
Y
i
❘
ρ
)
=
∏
i
=
1
M
∏
q
=
1
Q
ρ
q
Y
iq
and the probability of presence of an edge A ij between two nodes i and j, i being in the cluster q and j being in the cluster r, is only a function of the connection probability π qr between the clusters q and r.
3. The method of claim 2 , wherein in the generative model, the modeling of the at least one set of documents W ij d leads to the following joint distribution:
p
(
W
,
Z
,
θ
❘
A
,
Y
,
β
)
=
p
(
W
❘
A
,
Z
,
β
)
ρ
(
Z
❘
A
,
Y
,
θ
)
p
(
θ
)
=
∏
i
≠
j
M
{
∏
d
=
1
D
ij
∏
n
=
1
N
ij
d
∏
k
=
1
K
p
(
W
ij
dn
❘
β
k
)
Z
ij
dnk
}
A
ij
×
∏
i
≠
j
M
{
∏
d
=
1
D
ij
∏
α
=
1
N
ij
d
∏
q
,
r
Q
p
(
Z
ij
dn
❘
θ
qr
)
Y
iq
Y
jr
}
A
ii
∏
q
,
r
Q
Dir
(
θ
qr
;
α
)
.
wherein β k is an initial topic vector, a latent topic vector Z ij dn of the n-th word W ij dn follows a multinomial distribution whose parameters are only a function of q and r,
the vectors of topic proportions θ qr are drawn from a Dirichlet distribution having a parameter α>0, indicating a predominance of a plurality of topics over all pairs of clusters,
and the word W ij dn , given the topic vector Z ij dn of said word and for a given topic k of said topic vector, is drawn from a multinomial distribution whose parameters are only a function of topic k.
4. The method of claim 2 , wherein a complete data log likelihood of the generative model is:
log
p
(
A
,
W
,
Y
❘
ρ
,
π
,
β
)
=
log
∑
Z
∫
θ
p
(
A
,
W
,
Y
,
Z
,
θ
❘
ρ
,
π
,
β
)
d
θ
,
and an inference allows to maximize a decomposition function of the complete data log likelihood with the following lower bound:
log
p
(
A
,
W
,
Y
❘
ρ
,
π
,
β
)
=
ℒ
(
R
(
·
)
;
Y
,
ρ
,
π
,
β
)
+
KL
(
R
(
·
)
p
(
·
A
,
W
,
Y
,
ρ
,
π
,
β
)
)
,
(
6
)
where
ℒ
(
R
(
·
)
;
Y
,
ρ
,
π
,
β
)
=
∑
Z
∫
θ
R
(
Z
,
θ
)
x
×
log
p
(
A
,
W
,
Y
,
Z
,
θ
❘
ρ
,
π
,
β
)
R
(
Z
,
θ
)
d
θ
,
and KL denotes the Kullback-Leibler divergence between the true and approximate posterior distribution R (⋅) of (Z, θ), given the data and model parameters.
5. The method of claim 4 , further comprising a sub-step d)of the second step, wherein the sub-step d) comprises, for calculating the new estimated clustering Y:
a1. for the node i of the communications network, calculating a value of a lower bound L with a first estimated node clustering Y i , being considered in each cluster q, a second estimated node clustering Y j of the nodes j of the communications network being fixed,
determining the estimated clustering Y i , in which the lower bound L is a maximum, and storing a new estimated node clustering Y i , in a new estimated clustering Y,
a2. repeating sub-step a1, for each node i=1, . . . ,M, for obtaining the new estimated clustering Y.
6. The method of claim 4 , wherein the sub-step b4 is iterated with a variational expectation-maximization algorithm (VEM-algorithm), and the sub-step c is implemented with a Classification-VEM algorithm.
7. The method of claim 5 , wherein for the sub-steps a and b of the second step 2), the estimated probabilities π, β, ρ are given by maximizing the lower bound L, and are:
β
k
υ
∝
∑
i
≠
j
M
A
ij
∑
d
=
1
D
ij
∑
n
=
1
N
ij
dn
ϕ
ij
dnk
W
ij
dn
υ
,
∀
(
k
,
υ
)
,
ρ
q
∝
∑
i
=
1
M
Y
iq
,
∀
q
,
π
qr
=
∑
i
≠
j
M
Y
iq
Y
jr
A
ij
∑
i
≠
j
M
Y
iq
Y
jr
,
∀
(
q
,
r
)
.
8. The method of claim 1 , wherein the first step 1) comprises:
determining initial topics, by analyzing co-occurrence of words throughout the text data, with a variational expectation-maximization (VEM) for Latent Dirichlet Allocation (LDA), on each of the least one set of documents W ij d ;
calculating a topic matrix X comprising a plurality of topic matrix values each associated to a pair of nodes, wherein each of the topic matrix values corresponds to a major topic found in text data associated with the pair of nodes,
calculating a distance matrix, wherein Δ represents the distance matrix, the distance matrix Δ comprising a plurality of distance values each associated to a pair of nodes, wherein a distance value between nodes of the pair is calculated as follows:
Δ
(
i
,
j
)
=
∑
h
=
1
N
δ
(
X
ih
≠
X
jh
)
A
ih
A
jh
+
∑
h
=
1
N
δ
(
X
hi
≠
X
hj
)
A
hi
A
hj
,
performing a k-means clustering from said distance matrix Δ in order to obtain the clusters of first step 1).
9. The method of claim 1 , further comprising pre-processing received text data before sub-step b2 of the second step 2), wherein the pre-processing comprises at least one of:
deleting a word of the text data shorter than a predetermined word length,
deleting a word of the text data part of a predetermined list of stopwords,
deleting non-alphabetic characters in the text data,
detecting in the text data a plurality of words having a common root, or
selecting a reference word in the plurality of words, and replacing at least one word of said plurality of words by the reference word.
10. The method of claim 5 , wherein the convergence criterion of the third step 3) is fulfilled whenever a difference in absolute value between the lower bound value for the current iteration of third step 3) and a previously obtained lower bound value for a previous iteration of third step 3), is below a predetermined threshold.
11. The method of claim 1 , wherein the communications network is a telecommunication network, a computer network, the internet, a telephone network, or a TCP/IP network.
12. The method of claim 11 , wherein when the communications network is an email communication network, each node being one email address, and the at least one set of documents W ij d associated with the nodes i,j are emails sent in the communications network from first email address of the node i to a second email address of the node j.
13. The method of claim 11 , wherein the communications network is a Message Service (SMS/MMS) communication network, each node being one mobile phone number, and the at least one set of documents W ij d associated with two nodes i,j are messages sent in the communications network from a first mobile phone number of the node i to a second mobile phone number of the node j.
14. The method of claim 1 , wherein the communications network is a sensors communication network, the seconds communication network being as wireless sensors network, each node being one sensor, and the at least one set of documents W ij d associated with two nodes i,j are sent in the communications network from a first sensor of the node i to a second sensor of the node j.
15. The method of claim 5 , wherein steps 2) to 3) are carried out based on different pairs of fixed parameters (Q, K), the method further comprising:
5) a fifth step of calculating a quality score, based on an Integrated Completed Likelihood (ICU) criterion adapted to the generative model of claim 1 ,
6) a sixth step of repeating steps 1) to 5) for a plurality of different pairs of numbers (Q, K),
7) a seventh step of selecting, among the plurality different pairs of numbers (Q, K) that have been tested, an optimal pair of numbers (Q*,K*) for which a maximum quality score has been obtained.
16. The method of claim 1 , further comprising displaying a visual representation of the communications network, wherein:
a cluster is represented by a vertex having a size depending of the size of the cluster, and
an aggregate is represented by an arrow associated with two vertices, wherein an edge associated with a pair of a first vertex representing a first cluster and a second vertex representing a second cluster has a thickness that depends on a probability that a node included in the first cluster and a node included in the second cluster are associated with a text datum in the communications network.
17. A computer-implemented method for clustering nodes i,j which jointly generates a set of documents or datasets, a communications network comprising a number M of nodes, nodes i,j being associated with edges A ij of an adjacency matrix A,
a document edge W ij of text data by edge A ij , the document edge W ij being a vector W ij =(W ij 1 , . . . , W ij d , . . . , W ij Dij ), wherein D ij is a number of documents shared by or resulting of the interaction between the node i and the node j, where W ij d is at least one set of documents and W ij d includes text of a d th document shared by or resulting of the interaction between the node i and the node j,
the method comprising steps of:
1) a first step of initially clustering all the number M of nodes into a predetermined number Q of clusters, for determining a first estimated clustering Y=(Y 1 , Y 2 , . . . , Y M );
2) a second step of inference of a generative model,
wherein edges A ij are modeled with a Stochastic Block Model (SBM) and, conditionally to an estimated clustering Y, the sets of documents W ij d between and within the clusters are modeled according to a generative model of documents,
the second step of inference comprising the following sub-steps:
a. for a cluster q, determining from the first estimated clustering Y, an estimated proportion ρ q of each cluster q, an estimated probability π qr that a first node included in the cluster q and a second node included in a cluster r are associated with text data in the communications network;
b. determining from the first estimated clustering Y, an estimated distribution R (Z,θ), and an estimated probability β, with:
Z ij dn is a topic vector indicating a fist topic of the n-th word n of the at least one set of documents W ij d , wherein W ij dn ,
θ qr is a vector of topic proportions, wherein θ qrk is an estimated probability of a topic k of text data associated between the cluster q and the cluster r, making for a vector of topic proportions θ qr ,
βkv is a probability of each word v of dictionary being used in the topic k;
the sub-step b) comprising the following sub-steps:
b1. for each cluster q, aggregating the at least one set of documents W ij d of text data associated with all the pair of nodes (i,j) within the cluster q, so as to produce a first aggregate of text data associated with the cluster q,
and for each cluster q, aggregating the at least one set of documents W ij d associated with all the pair of nodes (i, j), from which the nodes i belongs to the cluster q and the node j belongs to the cluster r, so as to produce at least one second aggregate associated to a pair of clusters (q,r),
b2. processing the first and second aggregates to determine the topics k corresponding to text data of the first and second aggregates, with a predetermined number K of topics being determined, and
b3. processing the text data of the first and second aggregates to determine a distribution R(Z,θ) and a probability β;
b4. iteratively repeating sub-step b1, b2 and b3 of second step 2);
c. calculating a new estimated clustering Y of the nodes into Q clusters, in accordance with calculated probabilities β, ρ, π and the estimated distribution R(Z,θ) of the sub-steps a and b of the second step,
by determining, for each node i of the communications network, which optimal cluster provides a maximum value of a statistical criterion when each node i is assigned to said cluster, and assigning at least one node i to the determined optimal cluster;
3) a third step in which the second step 2) is repeated, with the new estimated clustering Y calculated in sub-step c and injected in the step 2), until a convergence criterion is fulfilled for determining and outputting a final clustering Y, a final proportion ρ q , a final distribution R(Z,θ), at least one final probability π qr , and a final probability βkv.
18. The method of claim 17 , wherein in the generative model, the modeling of the edge A ij leads to the following joint distribution:
p
(
A
,
Y
❘
ρ
,
π
)
=
p
(
A
❘
Y
,
π
)
p
(
Y
❘
ρ
)
,
where
p
(
A
❘
Y
,
π
)
=
∏
i
≠
j
M
p
(
A
ij
❘
Y
i
,
Y
j
,
π
)
=
∏
i
≠
j
M
∏
q
,
l
Q
p
(
A
ij
❘
π
qr
)
Y
iq
Y
jr
,
and
p
(
Y
❘
ρ
)
=
∏
i
=
1
M
p
(
Y
i
❘
ρ
)
=
∏
i
=
1
M
∏
q
=
1
Q
ρ
q
Y
iq
.
and the probability of presence of an edge A ij between two nodes i and j, i being in the cluster q and j being in the cluster r, is only a function of the connection probability π qr between the cluster q and r.
19. The method of claim 18 , wherein in the generative model, the modeling of the at least one set of documents W ij d leads to the following joint distribution:
p
(
W
,
Z
,
θ
❘
A
,
Y
,
β
)
=
p
(
W
❘
A
,
Z
,
β
)
ρ
(
Z
❘
A
,
Y
,
θ
)
p
(
θ
)
=
∏
i
≠
j
M
{
∏
d
=
1
D
ij
∏
n
=
1
N
ij
d
∏
k
=
1
K
p
(
W
ij
dn
❘
β
k
)
Z
ij
dnk
}
A
ij
×
∏
i
≠
j
M
{
∏
d
=
1
D
ij
∏
n
=
1
N
ij
d
∏
q
,
r
Q
p
(
Z
ij
dn
❘
θ
qr
)
Y
hi
Y
jr
}
A
ij
∏
q
,
r
Q
Dir
(
θ
pr
;
α
)
,
wherein β k is an initial topic vector, a topic vector Z ij dn of the n-th word W ij dn follows a multinomial distribution whose parameters are only a function of q and r,
the vectors of topic proportions θ qr are drawn from a Dirichlet distribution having a parameter α>0, indicating a predominance of a plurality of topics over all pairs of clusters,
and the word W ij dn given the topic vector Z ij dn of said word and for a given topic k of said topic vector, is drawn from a multinomial distribution whose parameters are only a function of topic k.
20. The method of claim 18 , wherein a complete data log likelihood of the generative model is:
log
p
(
A
,
W
,
Y
❘
ρ
,
π
,
β
)
=
log
∑
Z
∫
θ
p
(
A
,
W
,
Y
,
θ
❘
ρ
,
π
,
β
)
d
θ
,
and an inference allows to maximize a decomposition function of the complete data log likelihood with the following lower bound:
log
p
(
A
,
W
,
Y
❘
ρ
,
π
,
β
)
=
ℒ
(
R
(
·
)
;
Y
,
ρ
,
π
,
β
)
+
KL
(
R
(
·
)
p
(
·
A
,
W
,
Y
,
ρ
,
π
,
β
)
)
,
(
6
)
where
ℒ
(
R
(
·
)
;
Y
,
ρ
,
π
,
β
)
=
∑
Z
∫
θ
R
(
Z
,
θ
)
x
×
log
p
(
A
,
W
,
Y
,
Z
,
θ
❘
ρ
,
π
,
β
)
R
(
Z
,
θ
)
d
θ
,
and KL denotes the Kullback-Leibler divergence between the true and approximate posterior distribution R(⋅) of (Z,θ), given the data and model parameters.
21. The method of claim 20 , further comprising a step d) of the second step, wherein the step d) comprises, for calculating the new estimated clustering Y:
a1. for the node i of the network, calculating a value of a lower bound L q with a first estimated node clustering Y i , i being considered in each cluster q, a second estimated node clustering Y j of the nodes j of the communications network being fixed,
determining the estimated clustering Yi in which the lower bound L q is maximum, and storing a new estimated node clustering Y i , in a new estimated clustering Y,
a2. repeating sub-step a.1, for each node i=1, . . . , m, for obtaining the new estimated clustering Y.
22. The method of claim 17 , wherein the set is a co-authorship documents set, the nodes are at least one author or at least one groups of authors, and the at least one set of documents W ij d of text data associated with two nodes (i,j) is co-written by the two nodes (i,j).
23. The method of claim 17 , wherein when the set of documents is a set of written documents, the nodes are at least one persons or at least one groups of persons, and the set of documents W ij d associated with two nodes (i,j) includes a name of the at least one person or at least one group of persons corresponding to the two nodes (i,j).
24. The method of claim 23 , wherein the at least one person is a physical person or a corporation.
25. A non-transitory computer-readable storage device storing instructions executable by a processor for performing the method of clustering nodes of a communication network, the communications network comprising a number M of nodes, nodes i,j being associated with edges A ij of an adjacency matrix A, a message edge W ij of text data by edge A ij , the message edge W ij being a vector W ij =(W ij 1 , . . . , W ij d , . . . , W ij Dij ),wherein D ij is a number of text documents exchanged between the node i and the node j, where W ij d is at least one set of documents and W ij d includes text of a d th message exchanged between the node i and the node j, the method comprising:
1) initially clustering all the number M of nodes into clusters, for determining a first estimated clustering Y=(Y 1 , Y 2 , . . . , Y M ),;
2) a second step of inference of the generative model, wherein edges A ij are modeled with a Stochastic Block Model (SBM) and, conditionally to the an estimated clustering Y, the sets of documents W ij d between and within clusters are modeled according to a generative model,
the second step of inference comprising the following sub-steps:
a. for a cluster q, determining from the first estimated clustering Y, an estimated proportion ρ q of the cluster q, an estimated probability π qr that a first node included in the cluster q and a second node included in a cluster r are associated with text data in the communications network;
b. determining from the first estimated clustering Y, an estimated distribution R(Z,θ), and an estimated probability β, wherein:
Z ij dn is a topic vector indicating a first topic of the an n-th word n of the at least one set of documents W ij d , wherein W ij dn corresponds to the n-th word of a d-th document,
θ qr being a vector of topic proportions, wherein θ qrk is an estimated probability of a topic k of text data associated between the cluster q and the cluster r,
β kv a probability of each word v of dictionary being used in the topic k;
the sub-step b. comprising the following sub-steps:
b1. in the generative model, for each cluster q, aggregating the at least one set of documents W ij d associated with each of a pair of nodes (i,j) that are both within the cluster q, so as to produce a first aggregate of text data associated with the cluster q,
and for each cluster q, aggregating the at least one set of documents W ij d associated with each of the pairs of nodes (i,j), in which the node i belongs to the cluster q and the node j belongs to the cluster r, so as to produce at least one second aggregate associated to a pair of clusters (q,r),
b2. in the generative model, processing the first and second aggregates to determine the topic k corresponding to text data of the first and second aggregates, and
b3. processing the text data of the first and second aggregates to determine a distribution R(Z,θ) and a probability β;
b4. iteratively repeating sub-step b1, b2, b3 of the second step 2);
c. calculating a new estimated clustering Y, in accordance with calculated probabilities β, ρ, π and the estimated distribution R(Z,θ) of the sub-steps a and b of the second step,
by determining, for each node i of the communications network, which optimal cluster provides a maximum value of a statistical criterion when each node i is assigned to a cluster q, and assigning each node i to the determined optimal cluster;
3) a third step in which the second step 2) is repeated, with the new estimated clustering Y calculated in sub-step c and injected in the step 2), until a convergence criterion is fulfilled for determining and outputting a final clustering Y, at least one final proportions ρ q , a final distribution R(Z,θ), at least one final probability π qr , and at least one final probability β kv .