Training semantic image segmentation model comprising deformable convolutional neural network
View Patent ↗A method for training an image classification model includes obtaining first prediction class annotation information of a first image by using an image classification network based on a first model parameter of an offset network being fixed; determining a second model parameter corresponding to the image classification network by using a classification loss function based on the image content class information and the first prediction class annotation information; obtaining second prediction class annotation information of the first image by using the offset network based on the second model parameter of the image classification network being fixed; determining a third model parameter corresponding to the offset network by using the classification loss function based on the image content class information and the second prediction class annotation information; and training a semantic image segmentation network model based on the second model parameter and the third model parameter.
1 . A method for training an image classification model, performed by a computer device, and comprising:
obtaining a first image having class annotation information, the class annotation information representing image content class information of an image content that is included in the first image;
obtaining first prediction class annotation information of the first image by using an image classification network based on a first model parameter of an offset network being fixed, the offset network being configured to classify the first image based on an offset variable, the image classification network being configured to classify the image content in the first image;
determining a second model parameter corresponding to the image classification network by using a classification loss function based on the image content class information and the first prediction class annotation information;
obtaining second prediction class annotation information of the first image by using the offset network based on the second model parameter of the image classification network being fixed;
determining a third model parameter corresponding to the offset network by using the classification loss function based on the image content class information and the second prediction class annotation information; and
training a semantic image segmentation network model based on the second model parameter and the third model parameter, to obtain a final semantic image segmentation network model configured to determine a semantic segmentation result of a second image.
2 . The method according to claim 1 , wherein the determining the second model parameter comprises:
determining a prediction probability value corresponding to each class based on the image content class information and the first prediction class annotation information;
determining a classification loss of the classification loss function based on the prediction probability value corresponding to the each class; and
determining the second model parameter corresponding to the image classification network based on the classification loss of the classification loss function being minimum.
3 . The method according to claim 2 , wherein the classification loss function is represented as:
L
=
-
∑
k
=
1
N
I
(
c
=
k
)
log
P
c
,
wherein L represents the classification loss function, I( ) represents a Dirac function, N represents a total quantity of classes, c represents a c th class, k is greater than or equal to 1 and less than or equal to N, and P c represents a prediction probability value corresponding to the c th class.
4 . The method according to claim 1 , wherein the determining the third model parameter comprises:
determining a prediction probability value corresponding to each class based on the image content class information and the second prediction class annotation information;
determining a classification loss of the classification loss function based on the prediction probability value corresponding to the each class; and
determining the third model parameter corresponding to the offset network based on the classification loss of the classification loss function being maximum.
5 . The method according to claim 1 , wherein the obtaining the second prediction class annotation information comprises:
obtaining a feature image corresponding to the first image by using a deformable convolutional neural network, the deformable convolutional neural network being configured to predict the offset variable of the first image; and
obtaining the second prediction class annotation information corresponding to the feature image by using the offset network.
6 . The method according to claim 5 , wherein the obtaining the feature image comprises generating the feature image in the following manner:
y
(
p
0
)
=
∑
p
n
∈
R
w
(
p
n
)
x
(
p
0
+
p
n
+
Δ
p
n
)
,
wherein y(p 0 ) represents the feature image, p 0 represents a pixel value in the feature image, p n represents a position of a sampling point in a convolutional kernel, Δp n represents the offset variable, w(p n ) represents a weight value for performing a convolution in the convolutional kernel at a corresponding position in the first image, and x(p 0 +p n +Δp n ) represents a pixel value at the corresponding position in the first image.
7 . The method according to claim 1 , further comprising, after the determining the third model parameter:
obtaining third prediction class annotation information of the first image by using the image classification network based on the third model parameter corresponding to the offset network being fixed;
determining a fourth model parameter corresponding to the image classification network by using the classification loss function based on the image content class information and the third prediction class annotation information;
obtaining fourth prediction class annotation information of the first image by using the offset network based on the fourth model parameter of the image classification network being fixed; and
determining a fifth model parameter corresponding to the offset network by using the classification loss function based on the image content class information and the fourth prediction class annotation information; and
the training the semantic image segmentation network model comprises:
training the semantic image segmentation network model based on the second model parameter, the third model parameter, the fourth model parameter, and the fifth model parameter, to obtain the final semantic image segmentation network model.
8 . The method according to claim 1 , wherein the training the semantic image segmentation network model comprises:
determining an image content region corresponding to the first image based on an offset variable for training the offset network each time of training the offset network, the offset network being trained N times by using the second model parameter and the third model parameter, N being an integer greater than or equal to 1;
training the semantic image segmentation network model by using a target loss function based on the image content region; and
generating the semantic image segmentation network model based on a loss result of the target loss function being minimum.
9 . The method according to claim 8 , wherein the target loss function is represented as:
L
s
e
g
=
-
∑
k
=
1
N
∑
i
,
j
I
(
c
=
k
)
log
P
c
i
,
j
,
wherein L seg represents the target loss function, N represents a total quantity of classes, c represents a c th class, k is greater than or equal to 1 and less than or equal to N, I( ) represents a Dirac function,
P
c
i
,
j
represents a prediction probability value of the c th class at a pixel point, i represents a horizontal coordinate position of the pixel point in the first image, and j represents a vertical coordinate position of the pixel point in the first image.
10 . A server, comprising a memory, and a processor,
the memory being configured to store a program; and
the processor being configured to execute the program in the memory to perform operations of claim 1 .
11 . A non-transitory computer-readable storage medium, comprising instructions, the instructions, when run on a computer, causing the computer to perform the method according to claim 1 .
12 . An image processing method, performed by a computer device, and comprising:
obtaining a semantic segmentation result of an image by using a semantic image segmentation network model, the semantic image segmentation network model being obtained based on alternate training of an image classification network and an offset network, the offset network being configured to classify the image based on an offset variable, the image classification network being configured to classify image content in the image; and
processing the image based on the semantic segmentation result,
wherein during the alternate training of the image classification network and the offset network, the image classification network is trained by fixing a model parameter of the offset network and by using a classification loss function, and the offset network is trained by fixing a model parameter of the image classification network and by using the same classification loss function.
13 . A terminal device, comprising: a memory, and a processor,
the memory being configured to store a program; and
the processor being configured to execute the program in the memory to perform operations of claim 12 .
14 . A non-transitory computer-readable storage medium, comprising instructions, the instructions, when run on a computer, causing the computer to perform the method according to claim 12 .
15 . A model training apparatus, applicable to a computer device, and comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising:
first obtaining code configured to cause at least one of the at least one processor to obtain a first image having class annotation information, the class annotation information representing image content class information of an image content that is included in the first image;
second obtaining code configured to cause at least one of the at least one processor to obtain first prediction class annotation information of the first image by using an image classification network based on a first model parameter of an offset network being fixed, the offset network being configured to classify the first image based on an offset variable, the image classification network being configured to classify the image content in the first image;
first determining code configured to cause at least one of the at least one processor to determine a second model parameter corresponding to the image classification network by using a classification loss function based on the image content class information and the first prediction class annotation information;
third obtaining code configured to cause at least one of the at least one processor to obtain second prediction class annotation information of the first image by using the offset network based on the second model parameter of the image classification network being fixed;
second determining code configured to cause at least one of the at least one processor to determine a third model parameter corresponding to the offset network by using the classification loss function based on the image content class information and the second prediction class annotation information; and
training code configured to cause at least one of the at least one processor to train a semantic image segmentation network model based on the second model parameter and the third model parameter, to obtain a final semantic image segmentation network model that is used for determining a semantic segmentation result of a second image.
16 . The apparatus according to claim 15 , wherein the first determining code is further configured to cause at least one of the at least one processor to determine a prediction probability value corresponding to each class based on the image content class information and the first prediction class annotation information; determine a classification loss of the classification loss function based on the prediction probability value corresponding to the each class; and determine the second model parameter corresponding to the image classification network based on the classification loss of the classification loss function being minimum.
17 . The apparatus according to claim 16 , wherein the classification loss function is represented as:
L
=
-
∑
k
=
1
N
I
(
c
=
k
)
log
P
c
,
wherein L represents the classification loss function, I( ) represents a Dirac function, N represents a total quantity of classes, c represents a c th class, k is greater than or equal to 1 and less than or equal to N, and P c represents a prediction probability value corresponding to the c th class.
18 . The apparatus according to claim 15 , wherein the second determining code is further configured to cause at least one of the at least one processor to determine a prediction probability value corresponding to each class based on the image content class information and the second prediction class annotation information; determine a classification loss of the classification loss function based on the prediction probability value corresponding to the each class; and determine the third model parameter corresponding to the offset network based on the classification loss of the classification loss function being maximum.
19 . The apparatus according to claim 15 , wherein the third obtaining code is further configured to cause at least one of the at least one processor to obtain a feature image corresponding to the first image by using a deformable convolutional neural network, the deformable convolutional neural network being configured to predict the offset variable of the first image; and obtain the second prediction class annotation information corresponding to the feature image by using the offset network.
20 . The apparatus according to claim 19 , wherein the third obtaining code is further configured to cause at least one of the at least one processor to generate the feature image in the following manner:
y
(
p
0
)
=
∑
p
n
∈
R
w
(
p
n
)
x
(
p
0
+
p
n
+
Δ
p
n
)
,
wherein y(p 0 ) represents the feature image, p 0 represents a pixel value in the feature image, p n represents a position of a sampling point in a convolutional kernel, Δp n represents the offset variable, w(p n ) represents a weight value for performing a convolution in the convolutional kernel at a corresponding position in the first image, and x(p 0 +p n +Δp n ) represents a pixel value at the corresponding position in the first image.