Method and apparatus with degraded image restoration
A processor-implemented method with degraded image restoration includes: receiving a degraded training image; training a first teacher network of an image restoration network and a second teacher network of the image restoration network to infer differential images corresponding to the degraded training image, wherein each of the first teacher network and the second teacher network comprises a differentiable activation layer and a performance of the first teacher network is greater than that of the second teacher network; initially setting a student network of the image restoration network based on the second teacher network; and training the student network to infer a differential image corresponding to the degraded training image by iteratively backpropagating, to the student network, a contrastive loss that decreases a first difference between a third output of the student network and a first output of the first teacher network and increases a second difference between the third output and a second output of the second teacher network.
1 . A processor-implemented method with degraded image restoration, the method comprising:
training a first teacher network of an image restoration network and a second teacher network of the image restoration network to infer differential images corresponding to a degraded training image, wherein each of the first teacher network and the second teacher network comprises a differentiable activation layer and a performance of the first teacher network is greater than that of the second teacher network;
initially setting a student network of the image restoration network based on the second teacher network; and
training the student network to infer a differential image corresponding to the degraded training image by iteratively backpropagating, to the student network, a contrastive loss that decreases a first difference between a third output of the student network and a first output of the first teacher network and increases a second difference between the third output and a second output of the second teacher network.
2 . The method of claim 1 , wherein the training of the student network comprises iteratively backpropagating the contrastive loss to the student network by further applying, to the contrastive loss, a ground truth loss between the first output of the first teacher network and a ground truth corresponding to the degraded training image and one or more high-order derivative losses of a differential inference cost function corresponding to each of the first output and the ground truth.
3 . The method of claim 1 , wherein the first teacher network, the second teacher network, and the student network are trained based on a same one or more functions having a similar characteristic.
4 . The method of claim 1 , wherein the contrastive loss comprises a loss that, by the iterative backpropagation, decreases the first difference determined for each channel of the third output and the first output and increases the second difference determined for each channel of the third output and the second output.
5 . The method of claim 1 , wherein the training of the student network comprises iteratively backpropagating the contrastive loss to the student network by further applying, to the contrastive loss, either one or both of:
a first loss that decreases a first difference between the third output and a deblurred first output of a first deblurring teacher network identifying a blurred region corresponding to degradation information of each region of the degraded training image and increases a second difference between the third output and a deblurred second output of a second deblurring teacher network identifying the blurred region; and
a second loss that decreases a third difference between the third output and a denoised first output of a first denoising teacher network identifying a noisy region corresponding to degradation information of each region of the degraded training image and increases a fourth difference between the third output and a denoised second output of a second denoising teacher network identifying the noisy region.
6 . The method of claim 5 , wherein the degradation information comprises either one or both of noise and blur.
7 . The method of claim 1 , wherein the training of the student network comprises iteratively backpropagating the contrastive loss to the student network by further applying, to the contrastive loss, a first adversarial loss that is based on a degree of discrimination between the degraded training image and a backpropagated image obtained by backpropagating a first restored image generated by the student network based on the degraded training image and a second adversarial loss that is based on a degree of discrimination between a ground truth image corresponding to the degraded training image and a second restored image generated by the student network by backpropagating the ground truth image.
8 . The method of claim 7 , wherein the image restoration network further comprises:
a first discriminator configured to determine a difference between the degraded training image and the backpropagated image of the first restored image;
a second discriminator configured to determine a difference between the ground truth image and the second restored image; and
a backward network configured to backpropagate the first restored image or the ground truth image.
9 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1 .
10 . The method of claim 1 , wherein the initially setting of the student network based on the second teacher network comprises:
setting an initial value such that the student network has a same weight as that of the second teacher network; or
generating the student network by replicating the second teacher network.
11 . A processor-implemented method with degraded image restoration, the method comprising:
applying a learnable parameter to a generative network comprising a differentiable activation layer;
generating a restored image in which degradation information of a degraded image is restored by inputting the degraded image to the generative network to which the learnable parameter is applied; and
outputting the restored image,
wherein the learnable parameter comprises a second learnable parameter.
12 . The method of claim 11 , wherein the differentiable activation layer corresponds to the following equation, and is implemented by an exponential linear unit (ELU) activation layer comprising the second learnable parameter α,
ELU
(
x
)
=
{
x
,
if
x
>
0
α
*
(
exp
(
x
)
-
1
)
,
if
x
≤
0
.
Equation
13 . The method of claim 11 , wherein the differentiable activation layer corresponds to the following equation, and is implemented by a softplus activation layer comprising the second learnable parameter β,
Softplus
(
x
)
=
1
β
*
log
(
1
+
exp
(
β
*
x
)
)
.
Equation
14 . The method of claim 11 , wherein the learnable parameter comprises
a first parameter of a convolution layer comprised in the generative network.
15 . The method of claim 11 , wherein
the generative network is trained based on a contrastive loss that, by iterative backpropagation, decreases a first difference between a first output of a first teacher network and a third output of the generative network and increases a second difference between the third output and a second output of a second teacher network having performance lower than that of the first teacher network, and
the first teacher network and the second teacher network each comprises a differentiable activation layer, and are pre-trained to infer differential images corresponding to the degraded image.
16 . The method of claim 11 , wherein the degraded image is captured by an under-display camera (UDC).
17 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to generate a degraded image restorer, comprising a generative network comprising a differentiable activation layer, by performing the following operations, for the generative network,
wherein the operations comprise:
preparing a first teacher network and a second teacher network, wherein each of the first teacher network and the second teacher network comprises a differentiable activation layer and is pre-trained to restore a degraded training image, and a performance of the first teacher network is greater than that of the second teacher network;
initially setting a student network based on the second teacher network;
iteratively backpropagating, to the student network, a contrastive loss obtained from a loss function to update the generative network, wherein the contrastive loss is a loss that decreases a first difference between a third output of the student network and a first output of the first teacher network and increases a second difference between the third output and a second output of the second teacher network;
stopping the backpropagating when the loss function satisfies preset criteria; and
storing a set of parameters associated with the generative network in the non-transitory computer-readable storage medium.
18 . An electronic device comprising:
a memory configured to store therein a generative network comprising a differentiable activation layer;
one or more processors configured to execute executable instructions comprised in the memory;
and
a display,
wherein, when the instructions are executed in the one or more processors, the one or more processors configured to:
invoke a generative network comprising a differentiable activation layer;
apply a learnable parameter to the generative network; and
generate a restored image in which degradation information of a degraded image is restored by inputting the degraded image to the generative network to which the learnable parameter is applied,
wherein the learning parameter is a second parameter of the differentiable activation layer.
19 . The electronic device of claim 18 , wherein the differentiable activation layer corresponds to the following equation, and is implemented by an exponential linear unit (ELU) activation layer comprising a second learnable parameter α,
ELU
(
x
)
=
{
x
,
if
x
>
0
α
*
(
exp
(
x
)
-
1
)
,
if
x
≤
0
.
Equation
20 . The electronic device of claim 18 , wherein the differentiable activation layer corresponds to the following equation, and is implemented by a softplus activation layer comprising a second learnable parameter β,
Softplus
(
x
)
=
1
β
*
log
(
1
+
exp
(
β
*
x
)
)
.
Equation
21 . The electronic device of claim 18 , wherein the learnable parameter comprises:
a first parameter of a convolution layer comprised in the generative network.