Computer-implemented method, apparatus, and computer-program product

1. A computer-implemented method, comprising:

inputting a low-resolution image into a generator; and

generating a high-resolution image using the generator based on the low-resolution image;

wherein generating the high-resolution image comprises processing the low-resolution image through a plurality of super-resolution generating units arranged in series in the generator, wherein a respective output from a respective one of the plurality of super-resolution generating units has a respective increased image resolution as compared to a respective input to the respective one of the plurality of super-resolution generating units;

wherein the respective one of the plurality of super-resolution generating units comprises a super-resolution generating block and a bilinear interpolation block;

wherein generating the high-resolution image using the generator further comprises:

inputting a respective input to the super-resolution generating block and the bilinear interpolation block, respectively;

processing the respective input using the super-resolution generating block to generate a first intermediate output from the super-resolution generating block;

inputting the first intermediate output from the super-resolution generating block to the bilinear interpolation block; and

generating a second intermediate output using the bilinear interpolation block based on the respective input and the first intermediate output;

wherein the super-resolution generating block comprises a plurality of encoding modules and a plurality of decoding modules arranged in a u-net architecture;

a respective one of the plurality of encoding modules comprises a down-sampling convolutional layer, and a plurality of first residual blocks arranged in series; and

a respective one of the plurality of decoding modules comprises a plurality of second residual blocks arranged in series, and an up-sampling convolutional layer.

2. The computer-implemented method of claim 1 , wherein the respective one of the plurality of super-resolution generating units further comprises a concatenation block;

wherein generating the high-resolution image using the generator further comprises:

inputting the second intermediate output to the concatenation block; and

generating the respective output from the concatenation block based on the second intermediate output.

3. The computer-implemented method of claim 1 , wherein generating the high-resolution image using the generator further comprises:

inputting an output from a last one of the plurality of super-resolution generating units arranged in series into an additional super-resolution generating block; and

processing the output from the last one of the plurality of super-resolution generating units using the additional super-resolution generating block to generate the high-resolution image.

4. The computer-implemented method of claim 1 , the computer-implemented method further comprising using a feedback loop to train the generator based on a generator loss function.

5. The computer-implemented method of claim 4 , wherein the generator loss function is defined as:

L

total

=

∑

i

=

0

n

(

α

⁢

L

i

m

+

β

⁢

L

i

φ

)

;

wherein i stands for a respective number of a plurality of scales; L m is a mean square error function; L φ is a perceptual loss function; α stands for a weight coefficient for L m ; and β stands for a weight coefficients for L φ .

6. The computer-implemented method of claim 5 , wherein a neural network model is used to process a reference image and the high-resolution image;

L

m

=

min

⁢

∑

j

=

0

N

1

N

⁢



I

j

-

I

^

j



2

2

;

⁢

and

⁢

L

j

φ

(

I

~

,

I

)

=

1

C

j

⁢

H

j

⁢

W

j

⁢



φ

j

(

I

~

)

-

φ

j

(

I

)



2

2

;

wherein I stands for the reference image; Î stands for the high-resolution image; φ stands for the neural network model; j stands for a respective one of a plurality of output layers of the neural network model; C stands for a number of channels of a feature map outputted from the respective one of a plurality of output layers; H stands for a height of the feature map outputted from the respective one of a plurality of output layers; and W stands for a width of the feature map outputted from the respective one of a plurality of output layers.

7. The computer-implemented method of claim 6 , wherein the neural network model is a VGG-19 model.

8. The computer-implemented method of claim 4 , further comprising pre-training a convolutional neural network by:

inputting a plurality of training low-resolution images into the convolutional neural network;

inputting a plurality of training reference images into the convolutional neural network;

generating a plurality of training high-resolution images using the generator respectively based on the plurality of training low-resolution images;

computing loss for the plurality of training high-resolution images respectively based on the generator loss function; and

tuning the convolutional neural network based on the loss.

9. The computer-implemented method of claim 1 , the computer-implemented method further comprising:

inputting a reference image and the high-resolution image into a discriminator; and

verifying the high-resolution image using the discriminator based on the reference image;

wherein the generator and the discriminator are components of a generative adversarial network (GAN).

10. The computer-implemented method of claim 9 , wherein verifying the high-resolution image comprises processing the high-resolution image through a plurality of convolutional layers arranged in series, and a fully connection layer.

11. The computer-implemented method of claim 9 , the computer-implemented method further comprising using a feedback loop to mutually train the generator and the discriminator of the GAN, based on a generator loss function and a discriminator loss function.

12. The computer-implemented method of claim 11 , wherein the generator loss function is defined as:

min

⁢

V

⁡

(

D

,

G

)

=

∑

i

=

0

3

α

i

⁢

E

x

~

P

z

(

z

)

[

log

⁡

(

1

-

D

i

(

G

i

(

z

)

)

)

]

+

β

i

⁢

L

i

p

;

and

the discriminator loss function is defined as:

max

⁢

V

⁡

(

D

,

G

)

=

∑

i

=

0

3

α

i

⁢

E

x

~

P

data

(

x

)

[

log

⁡

(

D

i

(

x

)

)

]

+

E

x

~

P

z

(

z

)

[

log

⁡

(

1

-

D

i

(

G

i

(

z

)

)

)

]

;

wherein D stands for the discriminator; G stands for the generator; i stands for a respective number of a plurality of scales; α i and β i stand for weight coefficients corresponding to the respective number of a plurality of scales; E stands for a probability density distribution; L i p is a perceptual loss function; P data (x) stands for a probability distribution of the reference image; x stands for a sample from P data (x); P z (z) stands for a probability distribution of the high-resolution image; and z stands for a sample from P z (z).

13. The computer-implemented method of claim 12 , wherein a neural network model is used to process a reference image and the high-resolution image;

the perceptual loss function is calculated as follows:

L

j

φ

(

I

~

,

I

)

=

1

C

j

⁢

H

j

⁢

W

j

⁢



φ

j

(

I

~

)

-

φ

j

(

I

)



2

2

;

wherein I stands for the reference image; Î stands for the high-resolution image; φ stands for the neural network model; j stands for a respective one of a plurality of output layers of the neural network model; C stands for a number of channels of a feature map outputted from the respective one of a plurality of output layers; H stands for a height of the feature map outputted from the respective one of a plurality of output layers; and W stands for a width of the feature map outputted from the respective one of a plurality of output layers.

14. The computer-implemented method of claim 13 , wherein

the neural network model is a VGG-19 model.

15. The computer-implemented method of claim 13 , wherein

the neural network model is a sphereNet model.

16. The computer-implemented method of claim 11 , further comprising pre-training a convolutional neural network by:

inputting a plurality of training low-resolution images into the convolutional neural network;

inputting a plurality of training reference images into the convolutional neural network;

generating a plurality of training high-resolution images using the generator respectively based on the plurality of training low-resolution images;

computing generator loss for the plurality of training high-resolution images respectively based on the generator loss function;

computing discriminator loss for the plurality of training high-resolution images respectively based on the discriminator loss function; and

tuning the convolutional neural network based on the generator loss and the discriminator loss.

17. An apparatus, comprising:

a memory; and

one or more processors;

wherein the memory and the one or more processors are connected with each other; and

the memory stores computer-executable instructions for controlling the one or more processors to:

input a low-resolution image into a generator; and

generate a high-resolution image using the generator based on the low-resolution image by processing the low-resolution image through a plurality of super-resolution generating units arranged in series in the generator;

wherein a respective output from a respective one of the plurality of super-resolution generating units has a respective increased image resolution as compared to a respective input to the respective one of the plurality of super-resolution generating units;

wherein the respective one of the plurality of super-resolution generating units comprises a super-resolution generating block and a bilinear interpolation block;

wherein the memory further stores computer-executable instructions for controlling the one or more processors to:

input a respective input to the super-resolution generating block and the bilinear interpolation block, respectively;

process the respective input using the super-resolution generating block to generate a first intermediate output from the super-resolution generating block;

input the first intermediate output from the super-resolution generating block to the bilinear interpolation block; and

generate a second intermediate output using the bilinear interpolation block based on the respective input and the first intermediate output;

wherein the super-resolution generating block comprises a plurality of encoding modules and a plurality of decoding modules arranged in a u-net architecture;

a respective one of the plurality of encoding modules comprises a down-sampling convolutional layer, and a plurality of first residual blocks arranged in series; and

a respective one of the plurality of decoding modules comprises a plurality of second residual blocks arranged in series, and an up-sampling convolutional layer.

18. A computer-program product comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by a processor to cause the processor to perform:

inputting a low-resolution image into a generator; and

generating a high-resolution image using the generator based on the low-resolution image;

wherein generating the high-resolution image comprises processing the low-resolution image through a plurality of super-resolution generating units arranged in series in the generator, wherein a respective output from a respective one of the plurality of super-resolution generating units has a respective increased image resolution as compared to a respective input to the respective one of the plurality of super-resolution generating units;

wherein the respective one of the plurality of super-resolution generating units comprises a super-resolution generating block and a bilinear interpolation block;

wherein generating the high-resolution image using the generator further comprises:

inputting a respective input to the super-resolution generating block and the bilinear interpolation block, respectively;

processing the respective input using the super-resolution generating block to generate a first intermediate output from the super-resolution generating block;

inputting the first intermediate output from the super-resolution generating block to the bilinear interpolation block; and

generating a second intermediate output using the bilinear interpolation block based on the respective input and the first intermediate output;

wherein the super-resolution generating block comprises a plurality of encoding modules and a plurality of decoding modules arranged in a u-net architecture;

a respective one of the plurality of encoding modules comprises a down-sampling convolutional layer, and a plurality of first residual blocks arranged in series; and

a respective one of the plurality of decoding modules comprises a plurality of second residual blocks arranged in series, and an up-sampling convolutional layer.