Method for assessing aesthetic quality of natural image based on multi-task deep learning
The present application discloses a method for assessing aesthetic quality of a natural image based on multi-task deep learning. Said method includes: step 1: automatically learning aesthetic and semantic characteristics of the natural image based on multi-task deep learning; step 2: performing aesthetic categorization and semantic recognition to the results of automatic learning based on multi-task deep learning, thereby realizing assessment of aesthetic quality of the natural image. The present application uses semantic information to assist learning of expressions of aesthetic characteristics so as to assess aesthetic quality more effectively, besides, the present application designs various multi-task deep learning network structures so as to effectively use the aesthetic and semantic information for obtaining highly accurate image aesthetic categorization. The present application can be applied to many fields relating to image aesthetic quality assessment, including image retrieval, photography and album management, etc.
1. A method for assessing aesthetic quality of a natural image based on multi-task deep learning, comprising:
step 1: automatically learning aesthetic and semantic characteristics of a natural image based on multi-task deep learning;
step 2: performing aesthetic categorization and semantic recognition to results of automatic learning based on multi-task deep learning, thereby realizing assessment of aesthetic quality of the natural image,
wherein step 1 includes: automatically learning aesthetic characteristics by using the semantic information of the natural image based on multi-task deep learning; and
step 2 includes: performing aesthetic categorization and semantic recognition to the natural image based on multi-task deep learning by using the first equation:
θ
Λ
,
W
Λ
,
λ
Λ
=
arg
max
θ
,
W
,
λ
p
(
θ
,
W
,
λ
❘
X
,
Y
,
Z
)
wherein, X represents the natural image; Y represents an aesthetic category marker corresponding to the image; Z represents a semantic information marker corresponding to the image; θ represents a parameter that the aesthetic categorization task and the semantic recognition task have in common in the bottom layer of the multi-task deep learning network; W represents respective parameters for the aesthetic categorization task and the semantic recognition task in a higher layer of the multi-task deep learning network, W=[W a ,W s ]; W a represents a parameter specific to the aesthetic categorization task in the multi-task deep learning network; W s represents a parameter specific to the semantic recognition task in the multi-task deep learning network; λ represents a weight coefficient of the semantic recognition task in the joint learning; p(θ,W, λ|X,Y,Z) represents the posterior probability.
2. The method according to claim 1 , further comprising:
transforming the first equation into the following second equation according to Bayesian theory:
p (θ, W,λ|X,Y,Z )∞ p ( Y|X,θ,W a ) p ( Z|X,θ,W s ,λ) p (θ) p ( W ) p (λ)
wherein, p(Y|X,θ,W a ) represents a conditional probability of a corresponding aesthetic categorization task, p(Z|X,θ,W s ,λ) represents a conditional probability of a corresponding semantic recognition task, p(θ), p(W) and p(λ) are prior probabilities, respectively.
3. The method according to claim 1 , further comprising:
optimizing the first equation by means of:
λ
=
1
M
wherein M represents the number of categories of semantic annotations.
4. The method according to claim 2 , wherein the conditional probability of said aesthetic categorization task is solved by equation:
p
(
Y
❘
X
,
θ
,
W
a
)
=
∏
n
=
1
N
∑
c
=
1
C
1
{
y
n
=
c
}
p
(
y
n
=
c
❘
x
n
,
θ
,
W
a
)
wherein, N represents the number of all training samples, n represents the n th sample, n=1, 2, . . . N, C represents the number of categories of aesthetic quality, c represents the c th category, c=1, 2, . . . 0, 1{⋅} is an indicator function, when the variable in the braces is true, the value is 1, when it is false, the value is 0, y n represents the aesthetic category marker of the n th sample; and
x n represents the image data of the n th sample;
and the conditional probability p(y n =c|x n ,θ,W a ) of the n th sample is obtained by a softmax function in the multi-task deep learning network.
5. The method according to claim 2 , wherein the conditional probability of said semantic recognition task is solved by equation:
p
(
Z
❘
X
,
θ
,
W
S
,
λ
)
=
∏
n
=
1
N
∏
m
=
1
M
(
p
(
z
n
m
=
1
❘
x
n
,
θ
,
W
s
m
)
z
n
m
(
1
-
p
(
z
n
m
=
1
❘
x
n
,
θ
,
W
s
m
)
)
1
-
z
n
m
)
λ
wherein, M represents the number of all semantic attributes; m represents the m th semantic attribute, m=1, 2, . . . M; z n m represents the marker of the m th semantic attribute of the n th sample, whose value is 0 or 1; and W s m represents a network parameter corresponding to the m th semantic attribute;
and the conditional probability p(z n m =1|x n ,θ,W s m ) of the n th sample is obtained by a Sigmoid function in a multi-task deep learning network.