Fine-grained image recognition method and apparatus using graph structure represented high-order relation discovery
Embodiments of the present disclosure provides a fine-grained image recognition method and apparatus using graph structure represented high-order relation discovery, wherein the method includes: inputting an image to be classified into a convolutional neural network feature extractor with multiple stages, extracting two layers of network feature graphs in the last stage, constructing a hybrid high-order attention module according to the network feature graphs, and forming a high-order feature vector pool according to the hybrid high-order attention module, using each vector in the vector pool as a node, and utilizing semantic similarity among high-order features to form representative vector nodes in groups, and performing global pooling on the representative vector nodes to obtain classification vectors, and obtaining a fine-grained classification result through a fully connected layer and a classifier based on the classification vectors.
1. A fine-grained image recognition method using graph structure represented high-order relation discovery, comprising:
inputting an image to be classified into a convolutional neural network feature extractor with multiple stages, and extracting two layers of network feature graphs X i and Y i in the last stage;
constructing a hybrid high-order attention module enhanced by a space-gated network according to the network feature graphs X i and Y i and forming a high-order feature vector pool according to the hybrid high-order attention module;
using each vector in the high-order feature vector pool as a node to construct a graph neural network, and utilizing semantic similarity among high-order features to form representative vector nodes in groups; and
performing global pooling on the representative vector nodes to obtain classification vectors, and obtaining a fine-grained classification result through a fully connected layer and a classifier based on the classification vectors.
2. The method according to claim 1 , wherein the constructing a hybrid high-order attention module enhanced by a space-gated network according to the network feature graphs X i and Y i and forming a high-order feature vector pool according to the hybrid high-order attention module, comprises:
constructing an attention perception matrix P i corresponding to the hybrid high-order attention module, according to the network feature graphs X i and Y i ;
merging a spatial perception high-order matrix (P i X i T ) obtained according to the attention perception matrix P i and the network feature graph X i , a gated network and the network feature graph X i to obtain a position relation matrix E i ; and
transposing the position relation matrix E i and merging the transposed position relation matrix E i with an encoded network feature graph Y i to form the high-order feature vector pool.
3. The method according to claim 2 , wherein the constructing an attention perception matrix P i corresponding to the hybrid high-order attention module, according to the network feature graphs X i and Y i , comprises:
encoding the network feature graph Y i ;
determining the attention perception matrix P i of each position in the hybrid high-order attention module, according to the transposition of the network feature graph X i and the encoded network feature graph Y i :
P
i
=
𝒩
(
ℳ
(
1
C
∑
i
=
1
C
X
i
⊤
φ
(
Y
i
)
)
)
∈
ℝ
WH
×
WH
wherein, the W and H respectively represent width and height dimensions of the network feature graph, C represents a dimension of the network feature graph, (x)=sign(x)x −1/2 represents a matrix normalization method, sign represents symbol positive and negative characteristics of x, (x)=x/∥x∥ 2 2 represents L2 normalization, WH×WH represents a size dimension of the generated vector and T represents a transposition symbol.
4. The method according to claim 2 , wherein the merging a spatial perception high-order matrix (P i X i T ) obtained according to the attention perception matrix P i and the network feature graph X i , a gated network and the network feature graph X i to obtain a position relation matrix E i , comprises:
performing matrix multiplication on the attention perception matrix P i and the transposition of the network feature graph X i to obtain the spatial perception high-order matrix (P i X i T );
constructing the position relation matrix E i , according to the spatial perception high-order matrix (P i X i T ):
E
i
=
ℊ
(
∑
c
=
1
C
P
c
)
·
(
P
i
X
i
T
)
+
X
i
wherein, the gated network is a fully connected layer neural network of a learnable weight with a dimension of WH×1 .
5. The method according to claim 2 , wherein the transposing the position relation matrix E i and merging the transposed position relation matrix E i with an encoded network feature graph Y i to form the high-order feature vector pool, comprises:
transposing the position relation matrix E i , interacting the transposed position relation matrix E i with the encoded cross-layer matrix Y i , and averaging it in the WH dimension to obtain the high-order feature vector pool with a dimension of C N ×C :
𝒯
=
𝒩
(
ℳ
(
1
W
H
∑
i
=
1
W
H
φ
(
Y
i
)
T
E
i
)
)
∈
ℝ
C
N
×
C
wherein, C N represents the number of the elements and C represents the dimension of each vector.
6. The method according to claim 1 , wherein the using each vector in the high-order feature vector pool as a node to construct a graph neural network, and utilizing semantic similarity among high-order features to form representative vector nodes in groups, comprises:
using each vector in the high-order feature vector pool as one node to construct a first graph neural network, and constructing an adjacency matrix according to the semantic similarity to extract a graph embedding feature matrix;
determining a grouping mapping rule from high-dimensional nodes to low-dimensional nodes, according to a second graph neural network; and
grouping nodes according to the grouping mapping rule and the graph embedding feature matrix to obtain the representative vector nodes.
7. The method according to claim 6 , wherein the using each vector in the high-order feature vector pool as one node to construct a first graph neural network, and constructing an adjacency matrix according to the semantic similarity to obtain a graph embedding feature matrix, comprises:
decomposing vectors in the vector pool into C N vector combinations with a dimension of C, and using each vector with the dimension of C as one feature node to construct a first adjacency matrix A i,j of the first graph neural network:
A
i
,
j
=
τ
(
x
i
)
T
·
τ
(
x
j
)
τ
(
x
i
)
τ
(
x
j
)
wherein, A i,j represents a connection relationship between the i-th node and the j-th node, and τ(x i ) represents encoding a vector x i in the vector pool using τ coding;
adding a self-loop operation to the first adjacency matrix A i,j to obtain a second adjacency matrix Ã;
the second adjacency matrix à is:
Ã=A i,j +I
wherein, I∈ C N ×C N is expressed as an identity matrix with a diagonal of 1;
constructing a likelihood-constrained graph fusion algorithm according to the second adjacency matrix to obtain a graph embedding feature matrix H:
H
=
Re
LU
(
D
~
-
1
2
A
~
D
~
-
1
2
TW
g
)
wherein, W g ∈ C×d h represents a learnable parameter of the first graph neural network, d h represents a feature dimension after network transformation, T represents a matrix form of the above-mentioned high-order feature vector pool, ReLU represents a rectified linear unit and
D
~
=
∑
j
A
~
i
,
j
represents a diagonal matrix.
8. The method according to claim 7 , wherein the determining a grouping mapping rule from high-dimensional nodes to low-dimensional nodes according to a second graph neural network, comprises:
according to the second graph neural network, mapping C N nodes to be C r nodes to obtain a mapping relationship C N ×d h → C r ×d h ;
wherein, a grouping mapping rule matrix G from high-dimensional nodes to low-dimensional nodes is:
G
=
Re
LU
(
D
~
-
1
2
A
~
D
~
-
1
2
HW
emb
)
wherein, W emb ∈ d h ×C r represents a learnable matrix of the second graph neural network.
9. The method according to claim 6 , wherein the grouping nodes according to the grouping mapping rule and the graph embedding feature matrix to obtain the representative vector nodes, comprises:
according to the grouping mapping rule, clustering the convolutional graph embedding feature matrix H into a new graph of C r nodes to form the representative vector nodes Z:
Z
=
H
T
e
G
i
,
j
∑
j
=
1
C
r
e
G
i
,
j
∈
ℝ
d
h
×
C
r
.
10. A fine-grained image recognition device using graph structure represented high-order relation discovery, comprising:
at least one processor; and
a memory storing computer-executable instructions, wherein the computer-executable instructions are executed by the at least one processor, to enable the at least one processor to:
input an image to be classified into a convolutional neural network feature extractor with multiple stages, and extract two layers of network feature graphs X i and Y i in the last stage;
construct a hybrid high-order attention module enhanced by a space-gated network according to the two layers of network feature graphs X i and Y i , and form a high-order feature vector pool according to the hybrid high-order attention module;
use each vector in the high-order feature vector pool as a node to construct a graph neural network, and utilize a semantic similarity among high-order features to form representative vector nodes in groups; and
perform global pooling on the representative vector nodes to obtain classification vectors, and obtain a fine-grained classification result through a fully connected layer and a classifier based on the classification vectors.
11. The device according to claim 10 , wherein the at least one processor is further enabled to:
construct an attention perception matrix P i corresponding to the hybrid high-order attention module, according to the network feature graphs X i and Y i ;
merge a spatial perception high-order matrix (P i X i T ) obtained according to the attention perception matrix P i and the network feature graph X i , a gated network and the network feature graph X i to obtain a position relation matrix E i ; and
transpose the position relation matrix E i and merge the transposed position relation matrix E i with an encoded network feature graph Y i to form the high-order feature vector pool.
12. The device according to claim 11 , wherein the at least one processor is further enabled to:
encode the network feature graph Y i ;
determine the attention perception matrix P i of each position in the hybrid high-order attention module, according to the transposition of the network feature graph X i and the encoded network feature graph Y i :
P
i
=
𝒩
(
ℳ
(
1
C
∑
i
=
1
C
X
i
T
φ
(
Y
i
)
)
)
∈
ℝ
W
H
×
W
H
wherein, the W and H respectively represent width and height dimensions of the network feature graph, C represents a dimension of the network feature graph, (x)=sign(x)x −1/2 represents a matrix normalization method, sign represents symbol positive and negative characteristics of x, (x)=x/∥x∥ 2 2 represents L2 normalization, WH×WH represents a size dimension of the generated vector and T represents a transposition symbol.
13. The device according to claim 11 , wherein the at least one processor is further enabled to:
perform matrix multiplication on the attention perception matrix P i and the transposition of the network feature graph X i to obtain the spatial perception high-order matrix (P i X i T );
construct the position relation matrix E i , according to the spatial perception high-order matrix (P i X i T ):
E
i
=
ℊ
(
∑
c
=
1
C
P
c
)
·
(
P
i
X
i
T
)
+
X
i
wherein, the gated network is a fully connected layer neural network of a learnable weight with a dimension of WH×1 .
14. The device according to claim 11 , wherein the at least one processor is further enabled to:
transpose the position relation matrix E i , interact the transposed position relation matrix E i with the encoded cross-layer matrix Y i , and average it in the WH dimension to obtain the high-order feature vector pool with a dimension of C N ×C :
𝒯
=
𝒩
(
ℳ
(
1
W
H
∑
i
=
1
W
H
φ
(
Y
i
)
T
E
i
)
)
∈
ℝ
C
N
×
C
wherein, C N represents the number of the elements and C represents the dimension of each vector.
15. The device according to claim 10 , wherein the at least one processor is further enabled to:
use each vector in the high-order feature vector pool as one node to construct a first graph neural network, and construct an adjacency matrix according to the semantic similarity to extract a graph embedding feature matrix;
determine a grouping mapping rule from high-dimensional nodes to low-dimensional nodes, according to a second graph neural network; and
group nodes according to the grouping mapping rule and the graph embedding feature matrix to obtain the representative vector nodes.
16. The device according to claim 15 , wherein the at least one processor is further enabled to:
decompose vectors in the vector pool into C N vector combinations with a dimension of C, and use each vector with the dimension of C as one feature node to construct a first adjacency matrix A i,j of the first graph neural network:
A
i
,
j
=
τ
(
x
i
)
T
·
τ
(
x
j
)
τ
(
x
i
)
τ
(
x
j
)
wherein, A i,j represents a connection relationship between the i-th node and the j-th node, and τ(x i ) represents encoding a vector x i in the vector pool using τ coding;
add a self-loop operation to the first adjacency matrix A i,j to obtain a second adjacency matrix Ã;
the second adjacency matrix à is:
Ã=A i,j +I
wherein, I∈ C N ×C N is expressed as an identity matrix with a diagonal of 1;
construct a likelihood-constrained graph fusion algorithm according to the second adjacency matrix to obtain a graph embedding feature matrix H:
H
=
Re
LU
(
D
~
-
1
2
A
~
D
~
-
1
2
TW
g
)
wherein, W g ∈ C×d h represents a learnable parameter of the first graph neural network, d h represents a feature dimension after network transformation, T represents a matrix form of the above-mentioned high-order feature vector pool, ReLU represents a rectified linear unit and
D
~
=
∑
j
A
~
i
,
j
represents a diagonal matrix.
17. The device according to claim 16 , wherein the at least one processor is further enabled to:
according to the second graph neural network, map C N nodes to be C r nodes to obtain a mapping relationship C N ×d h → C r ×d h ;
wherein, a grouping mapping rule matrix G from high-dimensional nodes to low-dimensional nodes is:
G
=
Re
LU
(
D
~
-
1
2
A
~
D
~
-
1
2
HW
emb
)
wherein, W emb ∈ d h ×C r represents a learnable matrix of the second graph neural network.
18. The device according to claim 15 , wherein the at least one processor is further enabled to:
according to the grouping mapping rule, cluster the convolutional graph embedding feature matrix H into a new graph of C r nodes to form the representative vector nodes Z:
Z
=
H
T
e
G
i
,
j
∑
j
=
1
C
r
e
G
i
,
j
∈
ℝ
d
h
×
C
r
.
19. A non-transitory computer-readable storage medium storing a computer program, wherein the computer program is executed by a computer to:
input an image to be classified into a convolutional neural network feature extractor with multiple stages, and extract two layers of network feature graphs X i and Y i in the last stage;
construct a hybrid high-order attention module enhanced by a space-gated network according to the two layers of network feature graphs X i and Y i , and form a high-order feature vector pool according to the hybrid high-order attention module;
use each vector in the high-order feature vector pool as a node to construct a graph neural network, and utilize a semantic similarity among high-order features to form representative vector nodes in groups; and
perform global pooling on the representative vector nodes to obtain classification vectors, and obtain a fine-grained classification result through a fully connected layer and a classifier based on the classification vectors.
20. The non-transitory computer-readable storage medium according to claim 19 , wherein the computer program is further executed by the computer to:
construct an attention perception matrix P i corresponding to the hybrid high-order attention module, according to the network feature graphs X i and Y i ;
merge a spatial perception high-order matrix (P i X i T ) obtained according to the attention perception matrix P i and the network feature graph X i , a gated network and the network feature graph X i to obtain a position relation matrix E i ; and
transpose the position relation matrix E i and merge the transposed position relation matrix E i with an encoded network feature graph Y i to form the high-order feature vector pool.