Electronic device and method with image processing through canonical space
Provided are an electronic devices and methods for processing an image through a canonical space, the method including receiving training images of a scene obtained from a first sensor, generating first conversion images by converting the training images into images corresponding to a scene obtained through a second sensor, generating second conversion images by converting a color value of the training images and a color value of the first conversion images, and training a canonical image generative model configured to convert the training images, the first conversion images, and the second conversion images into canonical space images of a canonical color space and a canonical illumination space.
1 . A processor-implemented method of training an image processing model, the method comprising:
receiving training images of a scene obtained from a first sensor;
generating first conversion images by converting the training images into images corresponding to the scene obtained through a second sensor;
generating second conversion images by converting a color value of the training images and a color value of the first conversion images; and
training a canonical image generative model configured to convert the training images, the first conversion images, and the second conversion images into canonical space images of a canonical color space and a canonical illumination space.
2 . The method of claim 1 , wherein the generating of the first conversion images comprises generating the first conversion images by maintaining the color value of the training images and converting a pixel value of the training images into a pixel value of the second sensor.
3 . The method of claim 1 , wherein the generating of the first conversion images comprises generating the first conversion images by maintaining a pixel value of the training images and converting the color value of the training images into a color value of the second sensor.
4 . The method of claim 1 , wherein the generating of the second conversion images comprises generating the second conversion images by changing a red (R) channel value of the first conversion images and a blue (B) channel value of the first conversion images and applying a uniformly-distributed single illumination to the first conversion images with the changed R channel value and the changed B channel value.
5 . The method of claim 1 , wherein the canonical image generative model includes a convolutional neural network model.
6 . The method of claim 1 , wherein the training for converting into the canonical space images comprises training the canonical image generative model to minimize a loss function determined based on a difference between a reference image and each of outputs obtained by inputting the training images, the first conversion images, and the second conversion images to the canonical image generative model.
7 . The method of claim 6 , wherein the reference image is a standard Red Green Blue (sRGB) image, and the reference image is obtained by the first sensor.
8 . The method of claim 6 , wherein the loss function applied to the training images, the first conversion images, and the second conversion images is the same.
9 . The method of claim 1 , wherein the training for converting into the canonical space images further comprises performing training based on a CIE XYZ coordinate value.
10 . The method of claim 1 , wherein the training for converting into the canonical space images further comprises training the canonical image generative model to minimize a loss function determined based on a difference between the training images and a reference image.
11 . The method of claim 1 , further comprising:
receiving the canonical space images;
augmenting an image effect of the canonical space images; and
training an image effect augmentation model configured to output target images based on the augmented canonical space images.
12 . The method of claim 11 , wherein the target images are standard Red Green Blue (sRGB) images.
13 . The method of claim 11 , wherein the target images are:
standard Red Green Blue (sRGB) images of a sensor different from the first sensor;
images with different sensitivity from the training images; or
images to which a filter different from a filter of the training images is applied.
14 . The method of claim 11 , further comprising training the image processing model through a conversion matrix or a convolutional neural network encoder and a convolutional neural network decoder.
15 . The method of claim 1 , further comprising:
receiving an input image;
inputting the input image into the canonical image generative model to obtain a canonical image of the canonical color space and the canonical illumination space; and
obtaining a result image, to which image effect augmentation processing is applied, by using an image effect augmentation model configured to take the canonical image as an input.
16 . The method of claim 15 , wherein the result image is a standard Red Green Blue (sRGB) image.
17 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .
18 . An electronic device comprising:
a first sensor configured to capture training images of a scene;
a processor configured to:
receive the training images;
generate first conversion images by converting the training images into images corresponding to the scene obtained through a second sensor;
generate second conversion images by converting a color value of the training images and a color value of the first conversion images; and
train a canonical image generative model configured to convert the training images, the first conversion images, and the second conversion images into canonical space images of a canonical color space and a canonical illumination space.
19 . The electronic device of claim 18 , wherein the processor is further configured to:
receive the canonical space images;
augment an image effect of the canonical space images; and
train an image effect augmentation model configured to output target images based on the augmented canonical space images.
20 . The electronic device of claim 18 , further comprising:
a memory configured to store instructions; and
the processor is configured to execute the instructions stored in the memory to configure the processor to:
receive an input image;
input the input image into the canonical image generative model to obtain the canonical image of the canonical color space and the canonical illuminance space; and
obtain a result image, to which image effect augmentation processing is applied, by using an image effect augmentation model configured to take the canonical image as an input.