Electronic device for upscaling image and method for controlling same
An example electronic device includes a memory; a display; and at least one processor operatively coupled to the memory and the display. The at least one processor may be configured to divide an input image into a plurality of divided images, acquire image characteristics included in each of the plurality of divided images, identify at least one deep learning model to process each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in a memory on the basis of the image characteristics, acquire a plurality of upscaled segmented images corresponding respectively to the plurality of divided images through the at least one deep learning model, merge the plurality of upscaled divided images to obtain an upscaled image, and display the upscaled image on the display.
1 . An electronic device comprising:
memory storing instructions;
a display; and
at least one processor, comprising processing circuitry, operatively connected with the memory and the display, wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to:
divide an input image into a plurality of divided images, wherein a first divided image the plurality of divided images includes a padding area which overlaps with an adjacent divided image, and a seize of the padding area is based on a complexity of the first divided image;
obtain an image characteristic included in each of the plurality of divided images;
identify, based on the image characteristic, at least one deep learning model to process each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in the memory;
obtain, through the at least one deep learning model, a plurality of upscaled divided images respectively corresponding to the plurality of divided images;
obtain an upscaled image by merging the plurality of upscaled divided images based on a border processing operation for upscaled images of the first divided image and the adjacent divided images using the padding area; and
display the upscaled image on the display.
2 . The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to identify at least one of a number of the plurality of divided images or a size of the plurality of divided images based on at least one of an available size of the memory or a size of the input image.
3 . The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to obtain, as the image characteristic, at least one of image complexity of each of the plurality of divided images or whether a face image is included.
4 . The electronic device of claim 3 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to:
identify, as a first deep learning model, a deep learning model to process a divided image not including a face image and having an image complexity equal to or larger than a set value among the plurality of dividing images;
identify, as a second deep learning model, a deep learning model to process a divided image not including a face image and having an image complexity less than the set value among the plurality of dividing images; and
identify, as a third deep learning model, a deep learning model to process a divided image including a face image among the plurality of dividing images.
5 . The electronic device of claim 1 , comprising a plurality of processors, each of the plurality of processors comprising processing circuitry and being configured to perform image upscaling through at least one of the plurality of deep learning models.
6 . The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to obtain an average of a pixel value of a padding area corresponding to the area of each of the plurality of upscaled divided images and a pixel value of a padding area corresponding to the area of a neighboring upscaled divided image of each of the plurality of upscaled divided images, as a pixel value of the area.
7 . The electronic device of claim 1 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to store, as metadata of the upscaled image, at least one of information about the image characteristic, information about the identified at least one deep learning model, or position information about the plurality of divided images.
8 . The electronic device of claim 1 , wherein the size of the padding area increases as a complexity of the first divided image increases.
9 . A method for controlling an electronic device, the method comprising:
dividing an input image into a plurality of divided images, wherein a first divided image of the plurality of divided images includes a padding area which overlaps with an adjacent divided image, and a size of the padding area is based on a complexity of the first divided image;
obtaining an image characteristic included in each of the plurality of divided images;
identifying, based on the image characteristic, at least one deep learning model to process each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in a memory;
obtaining, through the at least one deep learning model, a plurality of upscaled divided images respectively corresponding to the plurality of divided images;
obtaining an upscaled image by merging the plurality of upscaled divided images based on a border processing operation for upscaled images of the first divided image and the adjacent divided images using the padding area; and
displaying the upscaled image.
10 . The method of claim 9 , wherein the dividing includes identifying at least one of a number of the plurality of divided images or a size of the plurality of divided images based on at least one of an available size of the memory or a size of the input image.
11 . The method of claim 9 , wherein obtaining the image characteristic obtains, as the image characteristic, at least one of image complexity of each of the plurality of divided images or whether a face image is included.
12 . The method of claim 11 , wherein the identifying identifies, as a first deep learning model, a deep learning model to process a divided image not including a face image and having an image complexity equal to or larger than a set value among the plurality of divided images; identifies, as a second deep learning model, a deep learning model to process a dividing image not including a face image and having an image complexity less than the set value among the plurality of dividing images; and identifies, as a third deep learning model, a deep learning model to process a divided image including a face image among the plurality of divided images.
13 . The method of claim 9 , wherein obtaining the plurality of upscaled divided images is performed by each of a plurality of processors using at least one of the plurality of deep learning models when the electronic device includes the plurality of processors.
14 . The method of claim 9 , wherein obtaining the plurality of upscaled divided images comprising obtaining, as a pixel value of the area, an average of a pixel value of a padding area corresponding to the area of each of the plurality of upscaled divided images and a pixel value of a padding area corresponding to the area of a neighboring upscaled divided image of each of the plurality of upscaled divided images.
15 . The method of claim 9 , further comprising storing, as metadata of the upscaled image, at least one of information about the image characteristic, information about the identified at least one deep learning model, or position information about the plurality of divided images.
16 . The method of claim 9 , wherein the size of the padding area increases as a complexity of the first divided image increases.
17 . An electronic device comprising:
memory storing instructions;
a display; and
at least one processor, comprising processing circuitry, operatively connected with the memory and the display, wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to:
obtain an image characteristic for each area of an input image,
divide the input image into a plurality of divided images based on the image characteristic for each area, wherein a first divided image of the plurality of divided images includes a padding area which overlaps with an adjacent divided image, and a size of the padding area is based on a complexity of the first divided image,
identify at least one deep learning model to process of each of the plurality of divided images from among a plurality of deep learning models for upscaling stored in the memory based on the image characteristic for each area,
obtain a plurality of upscaled divided images respectively corresponding to the plurality of divided images through the at least one deep learning model,
obtain an upscaled image by merging the plurality of upscaled divided images based on a border processing operation for upscaled images of the first divided image and the adjacent divided images using the padding area, and
display the upscaled image on the display.
18 . The electronic device of claim 17 , wherein the instructions, when executed by the at least one processor, individually or collectively, cause the electronic device to:
obtain at least one of an image complexity of the input image or whether a face image is included as the image characteristic,
identify at least one of a size, shape, or number of each of the plurality of divided images based on the image complexity or whether the face image is included, and
divide the input image into the plurality of divided images based on at least one of the size, shape, or number of each of the plurality of divided images.