Method and apparatus for super resolution

1 . A method of performing super-resolution (SR), comprising:

determining a direction type of an input image based on a gradient of the input image;

obtaining a first intermediate image based on the direction type using a look-up table (LUT) that stores an SR operation result of a pixel value combination corresponding to a kernel set mapped to the determined direction type, wherein the first intermediate image is an SR image corresponding to the input image;

applying SR interpolation to the input image to obtain a baseline image; and

obtaining an output image based on the first intermediate image and the baseline image, which comprises: obtaining an intermediate output image based on the baseline image and the first intermediate image; obtaining a second intermediate image by applying the input image to an SR neural network; and obtaining the output image based on the second intermediate image and the intermediate output image.

2 . The method of claim 1 , wherein the obtaining of the first intermediate image comprises:

extracting a non-flat region from the input image based on the gradient; and

obtaining the first intermediate image corresponding to the non-flat region based on a value stored in the LUT corresponding to the non-flat region of the input image.

3 . The method of claim 1 , wherein the LUT comprises a plurality of SR operation results respectively corresponding to a plurality of kernels comprised in a plurality of kernel sets mapped to a plurality of preset direction types,

wherein the SR operation result comprises a pixel value of the SR image mapped to each of a plurality of pixel value combinations corresponding to a size of the kernel.

4 . The method of claim 1 , wherein the kernel set includes a plurality of kernels having different respective shapes.

5 . The method of claim 1 , wherein the obtaining of the second intermediate image comprises:

obtaining the second intermediate image by applying the input image and a tuning parameter for image processing to the neural network,

wherein the neural network is trained to output the SR image at a high resolution based on a low-resolution image and a tuning parameter corresponding to the low-resolution image.

6 . The method of claim 5 , wherein the tuning parameter comprises at least one of:

a value indicating a noise level of the input image;

a blurring kernel of the input image; or

a standard deviation of a Gaussian kernel for blurring the input image.

7 . The method of claim 1 , further comprising:

outputting, in real time, the intermediate output image through a camera preview; and

providing the output image as a captured image of the input image.

8 . The method of claim 1 , wherein the obtaining of the output image comprises:

obtaining the output image based on the second intermediate image corresponding to a non-flat region of the input image and the intermediate output image,

wherein the non-flat region of the input image comprises at least a portion of the input image extracted based on the gradient of the input image.

9 . The method of claim 1 ,

wherein the first intermediate image is a corrected SR image corresponding to the second intermediate image, and

wherein the LUT stores an SR correction result of the pixel value combination corresponding to the kernel set mapped to the determined direction type.

10 . The method of claim 1 , wherein the

output image is obtained by applying the intermediate output image to an SR correction neural network.

11 . The method of claim 1 , wherein the determining of the direction type of the input image comprises:

obtaining a differential element for each axis of the input image by applying a mask for a differential for each axis of the input image; and

determining one of a plurality of preset direction types as the direction type of the input image based on an angle of a gradient of the input image obtained from the differential element for each axis of the input image.

12 . The method of claim 1 , wherein the obtaining of the output image comprises:

computing a sum of a pixel value of the baseline image and a pixel value of the first intermediate image, wherein the output image is obtained based on the sum.

13 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method for performing super-resolution (SR), wherein the method comprises:

determining a direction type of an input image based on a gradient of the input image;

obtaining a first intermediate image based on the direction type using a look-up table (LUT) that stores an SR operation result of a pixel value combination corresponding to a kernel set mapped to the determined direction type, wherein the first intermediate image is an SR image corresponding to the input image;

applying SR interpolation to the input image to obtain a baseline image; and

obtaining an output image based on the first intermediate image and the baseline image,

wherein the determining of the direction type of the input image comprises:

obtaining a differential element for each axis of the input image by applying a mask for a differential for each axis of the input image; and

determining one of a plurality of preset direction types as the direction type of the input image based on an angle of a gradient of the input image obtained from the differential element for each axis of the input image.

14 . An apparatus for super-resolution (SR) image processing, the apparatus comprising:

at least one processor configured to:

determine a direction type of an input image based on a gradient of the input image;

obtain a first intermediate image based on the direction type using a look-up table (LUT) that stores an SR operation result of a pixel value combination corresponding to a kernel set mapped to the determined direction type, wherein the first intermediate image is an SR image corresponding to the input image;

apply SR interpolation to the input image to obtain a baseline image; and

obtain an output image based on the first intermediate image and the baseline image,

wherein the at least one processor is configured to obtain the first intermediate image by:

obtaining a second intermediate image by applying the input image to an SR neural network, wherein the first intermediate image is a corrected SR image corresponding to the second intermediate image, and wherein the LUT stores a SR correction result of the pixel value combination corresponding to the kernel set mapped to the determined direction type.

15 . The apparatus of claim 14 , further comprising:

a memory configured to store the LUT.

16 . The apparatus of claim 14 , wherein the processor is configured to obtain the first intermediate image by:

extracting a non-flat region from the input image based on the gradient; and

obtaining the first intermediate image corresponding to the non-flat region, based on a value stored in the LUT corresponding to the non-flat region of the input image.

17 . The apparatus of claim 14 , wherein the processor is configured to obtain the output image by:

obtaining an intermediate output image based on the baseline image and the first intermediate image; and

obtaining the output image based on the second intermediate image and the intermediate output image.

18 . The apparatus of claim 17 , wherein the processor is configured to obtain the second intermediate image by:

obtaining the second intermediate image by applying the input image and a tuning parameter for image processing to the neural network,

wherein the neural network is trained to output the SR image at a high resolution based on a low-resolution image and a tuning parameter corresponding to the low-resolution image.

19 . The apparatus of claim 14 , wherein the kernel set includes a plurality of kernels having different respective shapes.

20 . The apparatus of claim 14 , wherein the processor is configured to obtain the output image by:

obtaining an intermediate output image based on the baseline image and the first intermediate image; and

obtaining the output image by applying the intermediate output image to an SR correction neural network.