Image semantic segmentation algorithm and system based on multi-channel deep weighted aggregation
An image semantic segmentation algorithm and system based on multi-channel deep weighted aggregation where the image semantic segmentation algorithm is based on multi-channel deep weighted aggregation. The aggregation includes semantic features with definite class information in an image, transition semantic features between low-level semantic and high-level semantic, and semantic features of context logic relationship in an image are extracted by a low-level semantic channel, an auxiliary semantic channel and a high-level semantic channel, respectively. The aggregation further includes three different semantic features obtained in S1 are fused by weighted aggregation to obtain global semantic information of the image; S3, the semantic features output from respective semantic channels in S1 and the global semantic information in S2 are used to compute loss function for training.
1. A computer-implemented method for performing image semantic segmentation in complex real-world environments based on multi-channel deep weighted aggregation, comprising:
executing, by one or more processors, instructions stored on a non-transitory computer-readable storage medium, wherein the instructions cause the processor(s) to performing follow operations:
S1, semantic features with definite class information in an image, transition semantic features between low-level semantic and high-level semantic, and semantic features of context logic relationship in the image are extracted by a low-level semantic channel, an auxiliary semantic channel and a high-level semantic channel, respectively;
S2, three different semantic features obtained in S1 are fused by weighted aggregation to obtain global semantic information of the image;
S3, the semantic features output from respective semantic channels in S1 and the global semantic information in S2 are used to compute loss function for training, wherein, in S1:
a shallow convolution structure network is used to construct the low-level semantic channel for extracting low-level semantic information, a depthwise separable convolution structure network is used to construct an auxiliary semantic channel, and transition semantic information obtained from the auxiliary semantic channel is fed back to the high-level semantic channel;
a deep convolution structure network is used to construct the high-level semantic channel for extracting high-level semantic information; and a process of extracting the low-level semantic information by the shallow convolution structure network includes:
LS(I H*W) )=S3(S2(I H*W )));
wherein, LS(I H*W ) is a extraction process of the low-level semantic information, I H*W is input image array, and S is a convolution stride.
2. The computer-implemented method for performing image semantic segmentation in complex real-world environments based on multi-channel deep weighted aggregation according to claim 1 , wherein, the separable convolution structure network includes three auxiliary modules connected in series and an attention mechanism module located at the end, each module transfers the extracted auxiliary semantic information of different levels to the high-level semantic channel;
a process of extracting the transition semantic information by the auxiliary semantic channel includes:
AS(K m*n )=Atten(Aux(Aux(Aux(K m*n ))))
wherein, AS(x) is the extraction process of the transition semantic information, K m*n is a feature array input to the auxiliary semantic channel, Aux(x) is the auxiliary module, and Atten(x) is the attention mechanism module;
the auxiliary semantic information output at each stage of the auxiliary semantic channel includes:
Aux 1 (K m*n )=Aux(K m*n )
Aux 2 (K m*n )=Aux(Aux,(K m*n ))
Aux 3 (K m*n )=Aux(Aux 2 (K m*n ))
Aux 1 ′(K m*n )=Up(AS(K m*n ),4)
where Up(x,k) is upsampling of the attention mechanism module to the high-level semantic channel, x is input, k is upsampling multiple, and its selected value is 4.
3. The computer-implemented method for performing image semantic segmentation in complex real-world environments based on multi-channel deep weighted aggregation according to claim 2 , wherein, the high-level semantic channel includes a start module, feature aggregation module, and a semantic embedding module, and a process of extracting the high-level semantic information includes:
it fuses with the auxiliary semantic information provided by the auxiliary semantic channel by the start module and the feature aggregation module to extract deep semantic information of the image, and then the semantic embedding module integrates the auxiliary semantic channel with the deep semantic information of the high-level semantic channel so as to obtain the high-level semantic information.
4. The computer-implemented method for performing image semantic segmentation in complex real-world environments based on multi-channel deep weighted aggregation according to claim 3 , wherein, the start module comprises two different down-sampling methods, convolution and maximum pooling, and two output results are connected and output after convolution; the feature aggregation module comprises two feature aggregation sub-modules, deep convolution is used to deepen the number of feature layers, the aggregation process of the feature aggregation module comprises:
F
GB
(
K
m
*
n
′
,
s
1
,
s
1
′
)
=
F
G
(
FG
(
K
m
*
n
′
,
s
1
)
,
s
1
′
)
wherein, FGB(x) is reasoning process of the feature aggregation module, FG(x,s 1 ) is reasoning process of the feature aggregation sub-module,
K
m
*
n
′
is input feature array, stride s 1 =2,
s
1
=
1
′
;
the reasoning processes of the feature aggregation modules of the three auxiliary modules in the auxiliary semantic channel include:
FGB
1
(
K
m
*
n
)
=
F
G
B
(
S
T
(
K
m
*
n
)
+
A
u
x
1
(
K
m
*
n
)
+
Aux
1
′
(
K
m
*
n
)
,
s
1
,
s
1
′
)
FGB
2
(
K
m
*
n
)
=
F
G
B
(
F
G
B
1
(
K
m
*
n
)
+
A
u
x
2
(
K
m
*
n
)
,
s
1
,
s
1
′
)
FGB
3
(
K
m
*
n
)
=
F
G
B
(
F
G
B
2
(
K
m
*
n
)
+
A
u
x
3
(
K
m
*
n
)
,
s
1
,
s
1
′
)
the semantic embedding module uses global average pooling and jump connection structure to deeply fuse the high-level semantic channel and the auxiliary semantic information provided by the auxiliary semantic channel;
the semantic information extraction process of the high-level semantic channel includes:
AS(K m*n )=SEB(FGB3(K m*n ))
wherein, AS(x) is the semantic information extraction process of the high-level semantic channel, SEB(x) is the reasoning process of the semantic embedding module.
5. The computer-implemented method for performing image semantic segmentation in complex real-world environments based on multi-channel deep weighted aggregation according to claim 1 , wherein, the fusion of semantic features in S2 includes weighted aggregation and fusion of three semantic features,
the weighted aggregation includes: updating the low-level semantic channel, the auxiliary semantic channel, and the high-level semantic channel to obtain weighted weights of the semantic features ε 1 ε 2 and ε 3 respectively, and the updating of the weight is determined adaptively by the MIoU={MIoU 1 , MIoU 2 , MIoU 3 } measured by the three semantic features in a verification set, the MIoU is obtained on the verification set by using the network weights of the respective channels, and then the weighted weights of the three semantic features are updated by the MIoU values according to the following formulas:
ε
1
=
{
1
+
MIoU
,
MIoU
1
=
max
{
MIoU
}
1
,
min
{
MIoU
}
<
MIoU
1
<
max
{
MIoU
}
1
-
MIoU
,
MIoU
1
=
min
{
MIoU
}
ε
2
=
{
1
+
MIoU
,
MIoU
2
=
max
{
MIoU
}
1
,
min
{
MIoU
}
<
MIoU
2
<
max
{
MIoU
}
1
-
MIoU
,
MIoU
2
=
min
{
MIoU
}
ε
3
=
{
1
+
MIoU
,
MIoU
3
=
max
{
MIoU
}
1
,
min
{
MIoU
}
<
MIoU
3
<
max
{
MIoU
}
1
-
MIoU
,
MIoU
3
=
min
{
MIoU
}
weighting processes of three semantic features include:
{
ϕ
1
(
x
)
=
ε
1
*
Conv
(
x
)
ϕ
2
(
x
)
=
ε
2
*
sigmoid
(
x
)
ϕ
3
(
x
)
=
ε
3
*
Up
(
x
)
wherein, Up is an upsampling with a magnification of 2 , Up(x) represents the upsampling, Conv is convolution operation, the calculation of sigmoid (x) includes:
sigmoid
(
x
)
=
1
1
+
e
x
.
6. The computer-implemented method for performing image semantic segmentation in complex real-world environments based on multi-channel deep weighted aggregation according to claim 3 , wherein, the fusion of the three semantic features includes multiplying the weighted results obtained by the pairwise semantic channels and then performing 3*3 convolution with a stride of 1, the pairwise aggregation processes of three semantic features comprise:
{
S
A
1
(
x
)
=
C
o
n
v
1
(
ϕ
1
(
x
)
*
ϕ
3
(
x
)
)
S
A
2
(
x
)
=
C
o
n
v
1
(
ϕ
1
(
x
)
*
ϕ
2
(
x
)
)
S
A
3
(
x
)
=
C
o
n
v
1
(
ϕ
2
(
x
)
*
ϕ
3
(
x
)
)
wherein, SA1(x) SA2(x) and SA3 (x) are the pairwise aggregation processes of three semantic features;
a composite process of the three semantic features obtained from the three semantic channels comprises:
MSA(x)=SA 1 (x)+SA 2 (x)+SA 3 (x)
wherein, MSA (x) is the composite process of the three semantic features.
7. The computer-implemented method for performing image semantic segmentation in complex real-world environments based on multi-channel deep weighted aggregation according to claim 1 , wherein, in S3, the semantic features output from respective channels are upsampled, the upsampled results and the global semantic information of the image are used together to calculate loss for random gradient descent learning, the loss function includes calculation of weight, calculation of the loss function, and determination of the loss function in a comprehensive training mechanism, wherein the calculation of the weight includes:
α
=
(
n
+
1
)
-
2
(
n
+
1
)
+
2
=
n
-
1
n
+
3
wherein, α is the weight, n is the number of classes;
calculation of the loss function comprises:
CED(x,y)=CEL(x,y)+α*DL(x,y)
wherein, x is prediction data, y is real data, CEL(x, y) is Cross-Entropy loss function and DL(x, y) is Dice Loss loss function;
the determination of loss function in comprehensive training mechanism includes:
Loss
(
P
r
e
,
A
1
,
A
2
,
A
3
,
T
)
=
CED
(
P
r
e
,
T
)
+
∑
i
=
1
3
C
E
D
(
A
i
,
T
)
wherein, T is a real label, Pre is output label of the global semantic information, and A 1 A 2 and A 3 are the output labels of the low-level semantic channel, the auxiliary semantic channel and the high-level semantic channel, respectively.