Knowledge distillation-based system for learning of teacher model and student model
View Patent ↗Disclosed is a system for improving the detection performance of objects of interest in blurry images. According to an embodiment of the disclosed system for improving the detection performance of objects of interest in blurry images, the system includes a teacher network including a first base subnet, a first feature pyramid subnet and a first detection subnet; and a student network including a second base subnet, a second feature pyramid subnet, a deblurring subnet and a second detection subnet, wherein the teacher network provides hints to the student network.
1 . A system, comprising:
a teacher network including a first base subnet, a first feature pyramid subnet and a first detection subnet; and
a student network including a second base subnet, a second feature pyramid subnet, a deblurring subnet and a second detection subnet,
wherein the teacher network provides hints to the student network, and
wherein the deblurring subnet includes:
a 1×1 convolution kernel; and
a 3×3 deformable convolution kernel,
wherein the 3×3 deformable convolution kernel includes a 3×3 convolution kernel and an offset field, and
wherein the offset field is estimated through the 3×3 convolution kernel.
2 . The system according to claim 1 , wherein the first base subnet and the first feature pyramid subnet perform a process expressed as the following Equation 1, and
wherein the second base subnet and the second feature pyramid subnet perform a process expressed as the following Equation 2:
F
n
=
Δ
{
f
n
l
}
l
=
1
L
=
ε
(
x
n
)
(
Equation
1
)
F
~
n
=
Δ
{
f
~
n
l
}
l
=
1
L
=
ε
(
x
~
n
)
(
Equation
2
)
where F n denotes a first parameter derived through the Equation 1, {tilde over (F)} n denotes a second parameter derived through the Equation 2,
{
f
n
l
}
l
=
1
L
denotes an l-th feature extracted from the first feature pyramid subnet,
{
f
~
n
l
}
l
=
1
L
denotes an l-th feature extracted from the second feature pyramid subnet, ε(⋅) denotes a combined function of the base subnet and the feature pyramid subnet, x n denotes an n-th sharp image, and {tilde over (x)} n denotes an n-th blurry image.
3 . The system according to claim 2 , wherein the first detection subnet performs a process expressed as the following Equation 3, and
wherein the second detection subnet performs a process expressed as the following Equation 4:
Z
n
=
Δ
{
(
z
n
k
,
b
n
k
)
}
k
=
1
K
=
𝒟
(
F
n
)
(
Equation
3
)
Z
~
n
=
Δ
{
(
z
~
n
k
,
b
~
n
k
)
}
k
=
1
K
=
𝒟
(
ℋ
(
F
~
n
)
)
(
Equation
4
)
where Z n denotes a third parameter derived through the Equation 3, {tilde over (Z)} n denotes a fourth parameter derived through the Equation 4,
z
n
k
denotes a k-th classification logit before applying a softmax function in the teacher network,
z
~
n
k
denotes a k-th classification logit before applying the softmax function in the student network,
b
n
k
denotes an object bounding box measured in the n-th sharp image,
b
~
n
k
denotes an object bounding box measured in the n-th blurry image, (⋅) denotes a function for deriving the first and second detection subnets, and (⋅) denotes a function for deriving the deblurring subnet.
4 . The system according to claim 1 , wherein a total loss function of the system is expressed as Equation 5:
L total =L det +λ 1 L SG +λ 2 L deblur (Equation 5)
where L total denotes the total loss function, L det denotes a loss function of the first detection subnet and the second detection subnet, L SG denotes a parameter which controls smoothness of probability distribution, L deblur denote a loss function of the deblurring subnet, and each of λ 1 and λ 2 is 0.5.
5 . The system according to claim 4 , wherein the L deblur performs a process expressed as the following Equation 6:
L
deblur
=
∑
n
=
1
N
∑
l
=
1
L
f
_
n
l
-
f
n
l
1
(
Equation
6
)
where M denotes
∑
n
=
1
N
∑
k
=
1
K
w
n
k
,
f
n
l
denotes an object bounding box measured in the n-th sharp image,
f
¯
n
l
denotes an object bounding box measured in the n-th blurry image, and ∥⋅∥ l denotes “1-norm”.
6 . The system according to claim 4 , wherein the L SG is expressed as the following Equation 7:
L SG =L conf +L reg (Equation 7)
where L SG denotes a total loss function of self-guided learning, L conf denotes an objective function for transferring classification knowledge from sharp features to blurry features, and L reg denotes a regression loss function of self-guided learning.
7 . The system according to claim 6 , wherein the student network is supervised by soft labels from the teacher network, and
wherein the soft labels are calculated by a process of the following Equation 8:
p
n
,
c
k
=
exp
(
z
n
,
c
k
/
τ
)
∑
c
=
1
C
exp
(
z
n
,
c
k
/
τ
)
(
Equation
8
)
where
P
n
k
denotes a soft estimation obtained from the n-th sharp image,
z
n
k
denotes a k-th classification logit before applying a softmax function in the teacher network, and τ denotes a parameter which controls smoothness of probability distribution.
8 . The system according to claim 7 , wherein the L conf is expressed as the following Equation 9:
L
conf
=
-
1
M
∑
n
=
1
N
∑
k
=
1
K
(
w
n
k
+
v
n
k
)
CE
(
p
~
n
k
,
p
n
k
)
(
Equation
9
)
where M denotes
∑
n
=
1
N
∑
k
=
1
K
w
n
k
,
p
~
n
k
denotes a soft estimation obtained from the n-th blurry image, and CE(⋅,⋅) denotes a cross-entropy loss function.
9 . The system according to claim 6 , wherein the L reg is expressed as the following Equation 10:
L
reg
=
1
M
∑
n
=
1
N
∑
k
=
1
K
w
n
k
b
~
n
k
-
b
n
k
1
(
Equation
10
)
where M denotes
∑
n
=
1
N
∑
k
=
1
K
w
n
k
,
b
n
k
denotes an object bounding box
measured in the n-th sharp image,
b
~
n
k
denotes an object bounding box measured in the n-th blurry image, and ∥⋅∥ 1 denotes “1-norm”.
10 . A system, comprising:
a teacher network including a first base subnet, a first feature pyramid subnet and a first detection subnet; and
a student network including a second base subnet, a second feature pyramid subnet, a deblurring subnet and a second detection subnet,
wherein the teacher network provides hints to the student network,
wherein the first base subnet and the first feature pyramid subnet perform a process expressed as the following Equation 1, and
wherein the second base subnet and the second feature pyramid subnet perform a process expressed as the following Equation 2:
F
n
=
Δ
{
f
n
l
}
l
=
1
L
=
ℰ
(
x
n
)
(
Equation
1
)
F
~
n
=
Δ
{
f
~
n
l
}
l
=
1
L
=
ℰ
(
x
~
n
)
(
Equation
2
)
where F n denotes a first parameter derived through the Equation 1, {tilde over (F)} n denotes a second parameter derived through the Equation 2,
{
f
n
l
}
l
=
1
L
denotes an l-th feature extracted from the first feature pyramid subnet,
{
f
~
n
l
}
l
=
1
L
denotes an l-th feature extracted from the second feature pyramid subnet, ε(⋅) denotes a combined function of the base subnet and the feature pyramid subnet, x n denotes an n-th sharp image, and {tilde over (x)} n denotes an n-th blurry image.
11 . The system according to claim 10 , wherein the first detection subnet performs a process expressed as the following Equation 3, and
wherein the second detection subnet performs a process expressed as the following Equation 4:
Z
n
=
Δ
{
(
z
n
k
,
b
n
k
)
}
k
=
1
K
=
𝒟
(
F
n
)
(
Equation
3
)
Z
~
n
=
Δ
{
(
z
~
n
k
,
b
~
n
k
)
}
k
=
1
K
=
𝒟
(
ℋ
(
F
~
n
)
)
(
Equation
4
)
where Z n denotes a third parameter derived through the Equation 3, {tilde over (Z)} n denotes a fourth parameter derived through the Equation 4,
z
n
k
denotes a k-th classification logit before applying a softmax function in the teacher network,
z
~
n
k
denotes a k-th classification logit before applying the softmax function in the student network,
b
n
k
denotes an object bounding box measured in the n-th sharp image,
b
~
n
k
denotes an object bounding box measured in the n-th blurry image, (⋅) denotes a function for deriving the first and second detection subnets, and (⋅) denotes a function for deriving the deblurring subnet.
12 . The system according to claim 10 , wherein a total loss function of the system is expressed as Equation 5:
L total =L det +λ 1 L SG +λ 2 L deblur (Equation 5)
where L total denotes the total loss function, L det denotes a loss function of the first detection subnet and the second detection subnet, L SG denotes a parameter which controls smoothness of probability distribution, L deblur denote a loss function of the deblurring subnet, and each of λ 1 and λ 2 is 0.5.
13 . The system according to claim 12 , wherein the L deblur performs a process expressed as the following Equation 6:
L
deblur
=
∑
n
=
1
N
∑
l
=
1
L
f
_
n
l
-
f
n
l
1
(
Equation
6
)
where M denotes
∑
n
=
1
N
∑
k
=
1
K
w
n
k
,
f
n
l
denotes an object bounding box measured in the n-th sharp image,
f
_
n
l
denotes an object bounding box measured in the n-th blurry image, and ∥⋅∥ 1 denotes “1-norm”.
14 . The system according to claim 12 , wherein the L SG is expressed as the following Equation 7:
L SG =L conf +L reg (Equation 7)
where L SG denotes a total loss function of self-guided learning, L conf denotes an objective function for transferring classification knowledge from sharp features to blurry features, and L reg denotes a regression loss function of self-guided learning.
15 . The system according to claim 14 , wherein the student network is supervised by soft labels from the teacher network, and
wherein the soft labels are calculated by a process of the following Equation 8:
p
n
,
c
k
=
exp
(
z
n
,
c
k
/
τ
)
∑
c
=
1
C
exp
(
z
n
,
c
k
/
τ
)
(
Equation
8
)
where
p
n
k
denotes a soft estimation obtained from the n-th sharp image,
z
n
k
denotes a k-th classification logit before applying a softmax function in the teacher network, and τ denotes a parameter which controls smoothness of probability distribution.
16 . The system according to claim 15 , wherein the L conf is expressed as the following Equation 9:
L
conf
=
-
1
M
∑
n
=
1
N
∑
k
=
1
K
(
w
n
k
+
v
n
k
)
CE
(
p
~
n
k
,
p
n
k
)
(
Equation
9
)
where M denotes
∑
n
=
1
N
∑
k
=
1
K
w
n
k
,
p
~
n
k
denotes a soft estimation obtained from the n-th blurry image, and CE(⋅, ⋅) denotes a cross-entropy loss function.
17 . The system according to claim 14 , wherein the L reg is expressed as the following Equation 10:
L
reg
=
1
M
∑
n
=
1
N
∑
k
=
1
K
w
n
k
b
~
n
k
-
b
n
k
1
(
Equation
10
)
where M denotes
∑
n
=
1
N
∑
k
=
1
K
w
n
k
,
b
n
k
denotes an object bounding box measured in the n-th sharp image,
b
~
n
k
denotes an object bounding box measured in the n-th blurry image, and ∥⋅∥ 1 denotes “1-norm”.
18 . A system, comprising:
a teacher network including a first base subnet, a first feature pyramid subnet and a first detection subnet; and
a student network including a second base subnet, a second feature pyramid subnet, a deblurring subnet and a second detection subnet,
wherein the teacher network provides hints to the student network, and
wherein a total loss function of the system is expressed as Equation 5:
L total =L det +λ 1 L SG +λ 2 L deblur (Equation 5)
where L total denotes the total loss function, L det denotes a loss function of the first detection subnet and the second detection subnet, L SG denotes a parameter which controls smoothness of probability distribution, L deblur denote a loss function of the deblurring subnet, and each of λ 1 and λ 2 is 0.5.
19 . The system according to claim 18 , wherein the L deblur performs a process expressed as the following Equation 6:
L
deblur
=
∑
n
=
1
N
∑
l
=
1
L
f
_
n
l
-
f
n
l
1
(
Equation
6
)
where M denotes
∑
n
=
1
N
∑
k
=
1
K
w
n
k
,
f
n
l
denotes an object bounding box measured in the n-th sharp image,
f
_
n
l
denotes an object bounding box measured in the n-th blurry image, and ∥⋅∥ 1 denotes “1-norm”.
20 . The system according to claim 18 , wherein the L SG is expressed as the following Equation 7:
L SG =L conf +L reg (Equation 7)
where L SG denotes a total loss function of self-guided learning, L conf denotes an objective function for transferring classification knowledge from sharp features to blurry features, and L reg denotes a regression loss function of self-guided learning.