Image processing device and operating method thereof
An image processing device includes an image sensor including a unit block including a plurality of pixels arranged adjacent to each other, wherein a plurality of nano-posts are arranged on the unit block, and a processor that processes an input image acquired through the image sensor.
1 . An image processing device comprising:
an image sensor comprising a unit block comprising a plurality of pixels arranged adjacent to each other, and a plurality of nano-posts arranged on the unit block; and
a processor configured to process an input image acquired through the image sensor by: dividing the input image into a plurality of first sub-images of a Bayer pattern including pixels having a same parallax among the plurality of pixels included in the input image;
converting the plurality of first sub-images into a plurality of RGB demosaic images;
converting the plurality of RGB demosaic images into a plurality of YCbCr images comprising first luminance data and color difference data;
generating second luminance data by applying the first luminance data, from among the plurality of YCbCr images, to a super resolution algorithm;
generating third luminance data by upscaling the first luminance data;
generating fourth luminance data by performing a weighted sum operation on the second luminance data and the third luminance data based on an attention map generated by the super resolution algorithm;
acquiring a plurality of second YCbCr images by updating the first luminance data of the plurality of YCbCr images to the fourth luminance data; and
generating an output image by merging a plurality of second sub-images generated based on the plurality of second YCbCr images.
2 . The image processing device of claim 1 , wherein the super resolution algorithm is an algorithm trained through a generator that learns a data distribution of a plurality of pieces of luminance data and a discriminator that learns to distinguish luminance data of an original image from luminance data generated by the generator.
3 . The image processing device of claim 2 , wherein the generator comprises a shallow feature extraction module, a deep feature extraction module, and a reconstruction module, and the reconstruction module comprises a sub-pixel convolution layer.
4 . The image processing device of claim 3 , wherein the shallow feature extraction module comprises at least one convolution layer configured to extract features for a low-resolution image from the plurality of pieces of luminance data input to the generator.
5 . The image processing device of claim 3 , wherein the deep feature extraction module comprises at least one residual block and at least one convolution layer configured to extract features for an ultra-high-resolution image from the plurality of pieces of luminance data input to the generator.
6 . The image processing device of claim 3 , wherein the reconstruction module is configured to generate the second luminance data by decoding, through the sub-pixel convolution layer, information in which first information extracted through the shallow feature extraction module and second information extracted through the deep feature extraction module are encoded.
7 . The image processing device of claim 2 , wherein the processor is further configured to:
acquire first data by multiplying, by the attention map, the second luminance data generated by passing the first luminance data through the generator;
acquire second data by multiplying a value obtained by subtracting the attention map from 1 by the third luminance data; and
generate the fourth luminance data by summing the second data and the third data.
8 . The image processing device of claim 1 , wherein the attention map is generated by a texture identifier of the super resolution algorithm, the texture identifier being a network in which a data set for a plurality of textures have been trained, and the attention map comprises weight values of each of a plurality of labels labeled in the data set for the plurality of textures.
9 . The image processing device of claim 1 , wherein the unit block is arranged in a 2×2 matrix and comprises four pixels each comprising a color filter of a same color.
10 . The image processing device of claim 1 , wherein the unit block comprises a plurality of unit blocks, and
wherein the image sensor has a quad Bayer pattern array in which the plurality of unit blocks are arranged in a 2×2 matrix.
11 . The image processing device of claim 10 , wherein the quad Bayer pattern array has one of an RGGB pattern, a GBRG pattern, a GRBG pattern, or a BGGR pattern.
12 . The image processing device of claim 1 , wherein the processor is further configured to:
convert the plurality of second YCbCr images into a plurality of final RGB images;
convert the plurality of final RGB images into the plurality of second sub-images; and
generate the output image by merging the plurality of second sub-images.
13 . An operating method of an image processing device, the operating method comprising:
acquiring an input image through an image sensor including a unit block including a plurality of pixels arranged adjacent to each other, and a plurality of nano-posts arranged on the unit block;
dividing the input image into a plurality of first sub-images of a Bayer pattern comprising pixels having a same parallax among the plurality of pixels included in the input image;
converting the plurality of first sub-images into a plurality of RGB demosaic images;
converting the plurality of RGB demosaic images into a plurality of YCbCr images comprising first luminance data and color difference data;
generating second luminance data by applying the first luminance data, from among the plurality of YCbCr images, to a super resolution algorithm;
generating third luminance data by upscaling the luminance data;
generating fourth luminance data by performing a weighted sum operation on the second luminance data and the third luminance data based on an attention map generated by the super resolution algorithm;
acquiring a plurality of second YCbCr images by updating the first luminance data of the plurality of YCbCr images to the fourth luminance data; and
generating an output image by merging a plurality of second sub-images generated based on the plurality of second YCbCr images.
14 . The operating method of claim 13 , wherein the super resolution algorithm is an algorithm trained through a generator that learns a data distribution of a plurality of pieces of luminance data and a discriminator that learns to distinguish luminance data of an original image from luminance data generated by the generator.
15 . The operating method of claim 14 , wherein the generator comprises a shallow feature extraction module, a deep feature extraction module, and a reconstruction module, and the reconstruction module comprises a sub-pixel convolution layer.
16 . The operating method of claim 15 , wherein the shallow feature extraction module comprises at least one convolution layer configured to extract features for a low-resolution image from the plurality of pieces of luminance data input to the generator.
17 . The operating method of claim 15 , wherein the deep feature extraction module comprises at least one residual block and at least one convolution layer configured to extract features for an ultra-high-resolution image from the plurality of pieces of luminance data input to the generator.
18 . The operating method of claim 15 , wherein the reconstruction module configured to generate the second luminance data by decoding, through the sub-pixel convolution layer, information in which first information extracted through the shallow feature extraction module and second information extracted through the deep feature extraction module are encoded.
19 . The operating method of claim 14 , wherein the generating of the output image comprises:
converting the plurality of second YCbCr images into a plurality of final RGB images;
converting the plurality of final RGB images into the plurality of second sub-images; and
generating the output image by merging the plurality of second sub-images.
20 . The operating method of claim 13 , wherein the generating of the fourth luminance data comprises:
acquiring first data by multiplying, by the attention map, the second luminance data generated by passing the first luminance data through the generator;
acquiring second data by multiplying a value obtained by subtracting the attention map from 1 by the third luminance data; and
generating the fourth luminance data by summing the second data and the third data.