Method for processing input image, generating rendered image by rendering the input image based on light source color information and panoramic image

1 . A processor-implemented method comprising:

obtaining an input image;

predicting light source color information of a scene corresponding to the input image and predicting a panoramic image corresponding to the input image using an image processing model; and

generating a rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image.

2 . The method of claim 1 , wherein the predicting of the light source color information of the scene corresponding to the input image and the panoramic image corresponding to the input image using the image processing model comprises:

generating an encoded feature of the input image by encoding the input image using an encoding network of the image processing model; and

predicting the light source color information and the panoramic image based on the encoded feature using a first prediction network of the image processing model.

3 . The method of claim 2 , wherein the generating of the encoded feature of the input image by encoding the input image using the encoding network of the image processing model comprises:

obtaining a plurality of image blocks by dividing the input image;

generating encoded features of the plurality of image blocks by encoding each of the plurality of image blocks; and

generating an encoded feature of the input image based on the encoded features of the plurality of image blocks.

4 . The method of claim 3 , wherein the generating of the encoded features of the plurality of image blocks by encoding each of the plurality of image blocks comprises:

generating an initial encoded feature of each image block by encoding each of the plurality of image blocks; and

improving a feature of the initial encoded feature of each image block using a first attention network of the image processing model to generate the improved encoded feature of each image block as encoded features of the plurality of image blocks.

5 . The method of claim 3 , wherein the predicting of the light source color information comprises:

for each image block, predicting light source color information of a scene corresponding to the image block based on the encoded feature of the image block, and determining a reliability of the image block; and

obtaining the light source color information of the scene by pooling the light source color information corresponding to the image block based on the reliability of each image block.

6 . The method of claim 3 , wherein the predicting of the panoramic image comprises:

obtaining a reliability of each image block;

obtaining a high dynamic range (HDR) image corresponding to each image block by decoding each of the encoded feature of each image block; and

generating the panoramic image by pooling the HDR image corresponding to the image block based on the reliability of each image block.

7 . The method of claim 2 , wherein the generating of the encoded feature of the input image by encoding the input image using the encoding network of the image processing model comprises:

generating a global feature of the input image by encoding the input image using the encoding network;

generating a lighting feature and a scene feature by decoupling features from the global feature using a decoupling network of the image processing model; and

improving features of the lighting feature and the scene feature using a second attention network of the image processing model to generate a scene-based lighting feature as the encoded feature of the input image.

8 . The method of claim 1 , wherein the predicting of the light source color information of the scene corresponding to the input image and the panoramic image corresponding to the input image using the image processing model comprises:

generating an encoded feature of the input image by encoding the input image using an encoding network of the image processing model;

predicting the light source color information based on the encoded feature using a second prediction network of the image processing model; and

predicting the panoramic image by decoding the encoded feature using a decoding network of the image processing model.

9 . The method of claim 1 , wherein the generating of the rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image comprises:

generating the rendered image by rendering the input image based on the input image, the panoramic image, and virtual object related information; and

outputting the rendered image using a video see through (VST) device.

10 . The method of claim 1 , wherein the generating of the rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image comprises:

generating a panoramic image corresponding to a white light source by performing color correction on the panoramic image based on the light source color information using a white balance method;

generating a first rendered image by rendering the input image based on the panoramic image of the white light source and virtual object related information;

generating actual light source color information of the scene by mapping the light source color information;

generating a second rendered image by adjusting a color of the first rendered image using the actual light source color information; and

outputting a target image formed by superposing the input image and the second rendered image using an optical see through (OST) device.

11 . The method of claim 10 , wherein the target image is formed by superposing the input image and a prime number of the second rendered images.

12 . The method of claim 1 , further comprising predicting camera parameters related to capturing of the input image using the image processing model.

13 . The method of claim 1 , further comprising:

obtaining training data, wherein the training data comprises a limited view image with a different field of view under a same scene, a first panoramic image, and first light source color information corresponding to when the limited view image and the first panoramic image are captured;

predicting second light source color information of a scene corresponding to the limited view image and a second panoramic image corresponding to the limited view image using the image processing model; and

training the image processing model by adjusting network parameters of the image processing model based on the first light source color information, the second light source color information, the first panoramic image, and the second panoramic image.

14 . An electronic device comprising:

one or more processors configured to:

obtain an input image;

predict light source color information of a scene corresponding to the input image and predict a panoramic image corresponding to the input image using an image processing model; and

generate a rendered image by rendering the input image based on either one or both of the light source color information and the panoramic image.

15 . The electronic device of claim 14 , wherein, for the predicting of the light source color information and the panoramic image, the one or more processors are configured to:

generate an encoded feature of the input image by encoding the input image using an encoding network of the image processing model; and

predict the light source color information and the panoramic image based on the encoded feature using a first prediction network of the image processing model.

16 . The electronic device of claim 14 , wherein, for the predicting of the light source color information and the panoramic image, the one or more processors are configured to:

generate an encoded feature of the input image by encoding the input image using an encoding network of the image processing model;

predict the light source color information based on the encoded feature using a second prediction network of the image processing model; and

predict the panoramic image by decoding the encoded feature using a decoding network of the image processing model.

17 . The electronic device of claim 14 , wherein, for the generating of the rendered image, the one or more processors are configured to:

generate the rendered image by rendering the input image based on the input image, the panoramic image, and virtual object related information; and

output the rendered image using a video see through (VST) device.

18 . The electronic device of claim 14 , wherein, for the generating of the rendered image, the one or more processors are configured to:

generate a panoramic image corresponding to a white light source by performing color correction on the panoramic image based on the light source color information using a white balance method;

generate a first rendered image by rendering the input image based on the panoramic image of the white light source and virtual object related information;

generate actual light source color information of the scene by mapping the light source color information;

generate a second rendered image by adjusting a color of the first rendered image using the actual light source color information; and

output a target image formed by superposing the input image and the second rendered image using an optical see through (OST) device.

19 . The electronic device of claim 14 , wherein the one or more processors are configured to:

obtain training data comprising a limited view image with a different field of view under a same scene, a first panoramic image, and first light source color information corresponding to when the limited view image and the first panoramic image are captured;

predict second light source color information of a scene corresponding to the limited view image and a second panoramic image corresponding to the limited view image using the image processing model; and

train the image processing model by adjusting network parameters of the image processing model based on the first light source color information, the second light source color information, the first panoramic image, and the second panoramic image.

20 . A processor-implemented method comprising:

obtaining training data, wherein the training data comprises a limited view image with a different field of view under a same scene, a first panoramic image, and first light source color information corresponding to when the limited view image and the first panoramic image are captured;

predicting second light source color information of a scene corresponding to the limited view image and a second panoramic image corresponding to the limited view image using an image processing model; and

training the image processing model by adjusting network parameters of the image processing model based on the first light source color information, the second light source color information, the first panoramic image, and the second panoramic image.