Image acquisition apparatus for estimating illumination information by inputting multispectral images to a deep learning network and electronic apparatus including the same
An image acquisition apparatus includes: a multispectral image sensor configured to acquire images in at least four channels based on a second wavelength band of about 10 nm to about 1,000 nm; and a processor configured to estimate illumination information of the images by inputting the images of at least four channels to a deep learning network trained in advance, and convert colors of the acquired images using the estimated illumination information.
1 . An image acquisition apparatus comprising:
a multispectral image sensor configured to acquire images of at least four channels based on a wavelength band of 10 nm to 1,000 nm; and
a processor configured to estimate illumination information of the images by inputting the images of at least four channels to a deep learning network trained in advance, and convert colors of the acquired images using the estimated illumination information,
wherein the estimated illumination information comprises local illumination information estimated by dividing the images in units of patches, and selectively inputting at least one patch among the patches to the deep learning network, and wherein the at least one patch is selectively input to the deep learning network based on an object included in the at least one patch and a color histogram of the at least one patch.
2 . The image acquisition apparatus of claim 1 , wherein the multispectral image sensor is further configured to acquire images in 16 channels, and
the processor is further configured to input the images of 16 channels received from the multispectral image sensor to the deep learning network, to estimate the illumination information.
3 . The image acquisition apparatus of claim 2 , wherein the processor is further configured to generate images in 31 channels by interpolation using the acquired images of 16 channels, and input the images of 31 channels to the deep learning network, to estimate the illumination information.
4 . The image acquisition apparatus of claim 1 , wherein the illumination information comprises at least one of an illumination vector corresponding to an illumination spectrum intensity, an XYZ vector indicating an illumination color, an RGB vector indicating an illumination color, an illumination color temperature, or an index value indicating pre-stored illumination information.
5 . The image acquisition apparatus of claim 1 , wherein the processor is further configured to input, to the deep learning network, images sampled at given intervals in a wavelength band corresponding to the images of at least four channels or images obtained by normalizing the images of at least four channels.
6 . The image acquisition apparatus of claim 1 , wherein the deep learning network comprises at least three convolution layers, a rectified linear unit function that is an activation function, and a max pooling layer.
7 . The image acquisition apparatus of claim 1 , wherein the deep learning network is trained in advance using a loss function comprising an angular loss or error, an L1 loss, and an L2 loss.
8 . The image acquisition apparatus of claim 7 , wherein the angular loss is calculated by Equation 3 below:
Angular
Loss
=
a
cos
(
L
GT
·
L
estimation
norm
(
L
GT
)
norm
(
L
estimation
)
)
[
Equation
3
]
the L1 loss is calculated by Equation 4 below:
MAE
Loss
=
1
N
∑
i
N
❘
"\[LeftBracketingBar]"
L
GT
-
L
estimation
❘
"\[RightBracketingBar]"
[
Equation
4
]
the L2 loss is calculated by Equation 5 below:
MSE
Loss
=
1
N
∑
i
N
(
L
GT
-
L
estimation
)
2
[
Equation
5
]
where L GT is ground-truth illumination information on an input image, and L estimation is estimated illumination information on an output image.
9 . The image acquisition apparatus of claim 1 , wherein the processor is further configured to generate a composite image by multiplying pre-calculated composite illumination by a reflectance map, to provide training images for the deep learning network.
10 . The image acquisition apparatus of claim 9 , wherein the reflectance map is obtained based on illumination information in a controlled environment or illumination information in an image acquired under daytime sunlight.
11 . The image acquisition apparatus of claim 1 , further comprising an image sensor configured to acquire an image based on a first wavelength band,
wherein the processor is further configured to convert colors of the image acquired by the image sensor by using the estimated illumination information.
12 . The image acquisition apparatus of claim 1 , wherein the estimated illumination information comprises global illumination information estimated based on all the images input to the deep learning network.
13 . The image acquisition apparatus of claim 1 , wherein the estimated illumination information of the images comprises one of a final result obtained by adding up a global illumination estimation result and local illumination estimation results, a final result obtained by selectively adding up the global illumination estimation result and the local illumination estimation results, a final result obtained by weighting and adding up the global illumination estimation result and the local illumination estimation results, and a final result obtained by bandpass filtering the global illumination estimation result and the local illumination estimation results.
14 . The image acquisition apparatus of claim 1 , further comprising a separate deep learning network configured to estimate a reflectance map.
15 . The image acquisition apparatus of claim 14 , wherein the processor is configured to acquire color-converted images using the estimated illumination information and the estimated reflectance map.
16 . An electronic apparatus comprising the image acquisition apparatus of claim 1 .
17 . An image acquisition apparatus comprising:
a multispectral image sensor configured to acquire images of at least four channels based on a wavelength band of 10 nm to 1,000 nm; and
a processor configured to estimate illumination information of the images by inputting the images of at least four channels to a deep learning network trained in advance, and convert colors of the acquired images using the estimated illumination information,
wherein the processor is further configured to generate a composite image by multiplying pre-calculated composite illumination by a reflectance map, to provide training images for the deep learning network, and
wherein the pre-calculated composite illumination has a random illumination spectrum calculated by Equation 6 below:
Random illumination spectrum= a 1 L A +a 2 L D65 +a 3 L LED +a 4 L Flu [Equation 6]
where a1, a2, a3, and a4 are randomly generated weight values, and L A , L D65 , L LED , and L Flu are illumination spectrums of incandescent illumination, daytime illumination, LED illumination, and fluorescent illumination, respectively.
18 . A method of controlling an image acquisition apparatus, the method comprising:
acquiring images of at least four channels based on a second wavelength band of 10 nm to 1,000 nm;
estimating illumination information of the images by inputting the images of at least four channels to a deep learning network trained in advance; and
converting colors of the acquired images by using the estimated illumination information,
wherein the estimated illumination information comprises local illumination information estimated by dividing the images in units of patches, and selectively inputting at least one patch among the patches to the deep learning network, and wherein the at least one patch is selectively input to the deep learning network based on an object included in the at least one patch and a color histogram of the at least one patch.