Image signal processing method using neural network model and computing apparatus for performing the same
A method of processing an image by using a neural network model includes obtaining an image captured via an image sensor, identifying a shooting context of the image, selecting a neural network model included in at least one of an image reconstruction module or an image correction module according to the shooting context, and processing the image by using the selected neural network model.
1 . A method of image processing, the method comprising:
obtaining an image captured via an image sensor;
identifying a shooting context of the image;
selecting a neural network model included in at least one of an image reconstruction module or an image correction module according to the shooting context; and
processing the image by using the selected neural network model,
wherein the selecting of the neural network model comprises selecting a detailed component of the image reconstruction module based on a shooting condition,
wherein a neural network model included in the image correction module comprises a first neural network model and a second neural network model,
the first neural network model is a model trained to minimize a difference between an inference image output when an input image and a correction parameter are fed into the first neural network model and a label image corresponding to the correction parameter,
the label image corresponding to the correction parameter is an image obtained by correcting the input image by using at least one correction algorithm to which the correction parameter is applied,
the correction parameter fed into the first neural network model is a correction parameter inferred when the input image is fed into the second neural network model, and
the second neural network model is a model trained to minimize a difference between the correction parameter inferred by the second neural network model when the input image is fed thereinto and a correction parameter that causes the label image to have preset image characteristics.
2 . The method of claim 1 , wherein the shooting context comprises at least one of the shooting condition or a shooting mode,
the shooting condition comprises at least one of an International Standards Organization (ISO) value of the image, a time when the image is captured, or a location where the image is captured, and
the shooting mode comprises at least one of a normal shooting mode, a night shooting mode, or a zoom shooting mode.
3 . The method of claim 2 , wherein a type of the image sensor is selected from among a plurality of different types of image sensors, based on at least one of the shooting condition or the shooting mode.
4 . The method of claim 3 , wherein the selecting of the neural network model comprises:
selecting the detailed component of the image reconstruction module based on at least one of the shooting context or the selected type of the image sensor; and
selecting one of a plurality of neural network models included in the image correction module, based on at least one of the shooting context, the selected type of the image sensor, or the selected detailed component of the image reconstruction module.
5 . The method of claim 4 , wherein the selecting of the detailed component of the image reconstruction module comprises:
primarily selecting the detailed component, based on at least one of the shooting mode or the selected type of image sensor; and
maintaining or changing the primarily selected detailed component, based on the shooting condition,
wherein the detailed component of the image reconstruction module corresponding to the shooting mode or the type of the image sensor is predesignated.
6 . The method of claim 4 , wherein the selecting of the detailed component of the image reconstruction module comprises selecting one of the plurality of neural network models included in the image reconstruction module, based on an ISO value included in the shooting condition.
7 . The method of claim 6 , wherein the selected neural network model comprises a denoising function and is a neural network model trained by using an image containing noise corresponding to the ISO value.
8 . The method of claim 4 , wherein the selecting of the one of the plurality of neural network models included in the image correction module comprises selecting a neural network model corresponding to a combination of the selected type of the image sensor and the selected detailed component of the image reconstruction module.
9 . The method of claim 8 , wherein the selected neural network model is a neural network model trained to correct image characteristics according to the shooting context.
10 . A computing apparatus for processing an image signal by using a neural network model, the computing apparatus comprising:
a memory storing a program for processing an image signal; and
at least one processor configured to execute the program to obtain an image captured via an image sensor, identify a shooting context of the image, select a neural network model included in at least one of an image reconstruction module or an image correction module according to the shooting context, and process the image by using the selected neural network model,
wherein the selecting of the neural network model comprises selecting a detailed component of the image reconstruction module based on a shooting condition,
wherein a neural network model included in the image correction module comprises a first neural network model and a second neural network model,
the first neural network model is a model trained to minimize a difference between an inference image output when an input image and a correction parameter are fed into the first neural network model and a label image corresponding to the correction parameter,
the label image corresponding to the correction parameter is an image obtained by correcting the input image by using at least one correction algorithm to which the correction parameter is applied,
the correction parameter fed into the first neural network model is a correction parameter inferred when the input image is fed into the second neural network model, and
the second neural network model is a model trained to minimize a difference between the correction parameter inferred by the second neural network model when the input image is fed thereinto and a correction parameter that causes the label image to have preset image characteristics.
11 . The computing apparatus of claim 10 , wherein the shooting context comprises at least one of the shooting condition or a shooting mode,
the shooting condition comprises at least one of an International Standards Organization (ISO) value of the image, a time when the image is captured, or a location where the image is captured, and
the shooting mode comprises at least one of a normal shooting mode, a night shooting mode, or a zoom shooting mode.
12 . The computing apparatus of claim 11 , wherein a type of the image sensor is selected from among a plurality of different types of image sensors, based on at least one of the shooting condition or the shooting mode.
13 . The computing apparatus of claim 12 , wherein, in selecting the neural network model, the at least one processor is further configured to execute the program to, after selecting the detailed component of the image reconstruction module based on at least one of the shooting context or the selected type of the image sensor, select one of a plurality of neural network models included in the image correction module based on at least one of the shooting context, the selected type of the image sensor, or the selected detailed component of the image reconstruction module.
14 . The computing apparatus of claim 13 , wherein, in selecting the detailed component of the image reconstruction module, the at least one processor is further configured to execute the program to, after primarily selecting the detailed component based on at least one of the shooting mode or the selected type of the image sensor, maintain or change the primarily selected detailed component, based on the shooting condition, and
the detailed component of the image reconstruction module corresponding to the shooting mode or the type of the image sensor is predesignated.
15 . The computing apparatus of claim 13 , wherein, in selecting the detailed component of the image reconstruction module, the at least one processor is further configured to execute the program to select one of the plurality of neural network models included in the image reconstruction module, based on an ISO value included in the shooting condition.
16 . The computing apparatus of claim 15 , wherein the selected neural network model comprises a denoising function and is a neural network model trained by using an image containing noise corresponding to the ISO value.
17 . The computing apparatus of claim 13 , wherein, in selecting the one of the plurality of neural network models included in the image correction module, the at least one processor may be further configured to execute the program to select a neural network model corresponding to a combination of the selected type of the image sensor and the selected detailed component of the image reconstruction module.
18 . A non-transitory computer-readable recording medium having recorded thereon a program for performing an image processing method comprising:
obtaining an image captured via an image sensor;
identifying a shooting context of the image;
selecting a neural network model included in at least one of an image reconstruction module or an image correction module according to the shooting context; and
processing the image by using the selected neural network model,
wherein the selecting of the neural network model comprises selecting a detailed component of the image reconstruction module based on a shooting condition,
wherein a neural network model included in the image correction module comprises a first neural network model and a second neural network model,
the first neural network model is a model trained to minimize a difference between an inference image output when an input image and a correction parameter are fed into the first neural network model and a label image corresponding to the correction parameter,
the label image corresponding to the correction parameter is an image obtained by correcting the input image by using at least one correction algorithm to which the correction parameter is applied,
the correction parameter fed into the first neural network model is a correction parameter inferred when the input image is fed into the second neural network model, and
the second neural network model is a model trained to minimize a difference between the correction parameter inferred by the second neural network model when the input image is fed thereinto and a correction parameter that causes the label image to have preset image characteristics.