Method and device with image processing
View Patent ↗A device with image processing includes: an image acquirer configured to acquire a plurality of images each having a different brightness; and one or more processors configured to extract an illumination map for an input image of the images and an illuminant color of the input image from the input image and temporal correlation information of the plurality of images, based on an illumination extraction model.
1 . A device with image processing, the device comprising:
an image acquirer configured to acquire a plurality of images, each having a different brightness; and
one or more processors configured to, based on an illumination extraction model, extract an illumination map and an illuminant color for a single input image of the plurality of images, and generate temporal correlation information indicating illumination changes across the plurality of images,
wherein the generating of the temporal correlation information comprises compressing color channels of the plurality of images into a single channel representation that represents values of the color channels, and
wherein, for the generating of the temporal correlation information, the one or more processors are configured to:
reshape a dimension of the single channel into a different dimension; and
determine temporal correlation information from attention data generated from the reshaped single channel and the plurality of images.
2 . The device of claim 1 , wherein the image acquirer is configured to acquire the plurality of images including one or more images having a brightness different from that of the input image.
3 . The device of claim 1 , wherein, for the generating of the illumination map and the illuminant color for the input image, the one or more processors are configured to:
extract a color map for each color channel from the input image using one or more convolution layers of the illumination extraction model; and
determine an illuminant color vector indicating the illuminant color based on the extracted color map for each color channel and an illuminant confidence map.
4 . The device of claim 3 , wherein the one or more processors are configured to generate a reflectance map from the input image using the illumination map and the illuminant color vector.
5 . The device of claim 3 , wherein the one or more processors are configured to generate a temporal gradient map as the illuminant confidence map by accumulating a difference for each temporal frame pixel by pixel from the plurality of images.
6 . The device of claim 3 , wherein the one or more processors are configured to:
generate another illumination map using the input image and a reflectance map; and
generate illuminant correlation information between the illumination map and the other illumination map as the illuminant confidence map.
7 . The device of claim 1 , wherein the illumination extraction model includes a pyramid pooling layer of propagating, to a subsequent layer, output data in which a result of performing an individual convolution operation on data pooled with different sizes to input data is concatenated to the input data.
8 . The device of claim 1 , wherein the one or more processors are configured to generate a white balanced image from the input image using the extracted illuminant color.
9 . The device of claim 1 , wherein the one or more processors are configured to extract, from the input image, a reflectance map for a same temporal frame as a temporal frame of the illumination map based on a reflectance extraction model.
10 . The device of claim 9 , wherein the one or more processors are configured to share feature data extracted from at least a portion of layers of the reflectance extraction model with the illumination extraction model.
11 . The device of claim 1 , wherein the image acquirer is configured to acquire the plurality of images captured under an alternating current (AC) illuminant.
12 . The device of claim 1 , wherein the image acquirer is configured to acquire each of the plurality of images at a different exposure time.
13 . The device of claim 1 , wherein the processor is configured to:
generate a plurality of illumination maps corresponding to respective temporal frames from the plurality of images using the illumination extraction model;
generate a plurality of reflectance maps corresponding to the respective temporal frames from the plurality of images using a reflectance extraction model; and
generate a composite image from the plurality of illumination maps and the plurality of reflectance maps.
14 . The device of claim 13 , wherein the processor is configured to reconstruct a high dynamic range (HDR) image from the plurality of illumination maps and the plurality of reflectance maps based on an image fusion model.
15 . A method with image processing, the method comprising:
acquiring a plurality of images each having a different brightness;
generating, based on an illumination extraction model and generated temporal correlation information indicating illumination changes across the plurality of images, an illumination map and an illuminant color for a single input image of the plurality of images,
wherein the generating of the temporal correlation information comprises compressing color channels of the plurality of images into a single channel representation that represents values of the color channels, and
wherein the generating of the temporal correlation information comprises:
reshaping a dimension of the single channel into a different dimension; and
determining temporal correlation information from attention data generated from the reshaped single channel and the plurality of images.
16 . The method of claim 15 , wherein the generating of the illumination map and the illuminant color for the single input image comprises:
extracting a color map for each color channel from the input image using one or more convolution layers of the illumination extraction model; and
determining an illuminant color vector indicating the illuminant color based on the extracted color map for each color channel and an illuminant confidence map.
17 . The method of claim 15 , wherein the generating of the illumination map and the illuminant color for the single input image comprises propagating, from a pyramid pooling layer of the illumination extraction model to a subsequent layer, output data in which a result of performing an individual convolution operation on data pooled with different sizes to input data is concatenated to the input data.
18 . The method of claim 15 , further comprising training the illumination extraction model based on a loss determined based on either one or both of the illumination map and the illuminant color.
19 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 15 .
20 . A method with image processing, the method comprising:
generating, based on a generated temporal correlation information, an illumination map and an illuminant color of a single input image among a plurality of images, each having a different brightness, using an illumination extraction model;
determining an illuminant color vector of the illuminant color based on a color map extracted for each color channel from the input image using a portion of the illumination extraction model;
generating a reflectance map of the input image based on the illumination map and the illuminant color vector; and
generating a white balanced image of the input image based on the illumination map and the reflectance map,
wherein the generating of the temporal correlation information comprises:
reshaping a dimension of the single channel into a different dimension; and
determining temporal correlation information from attention data generated from the reshaped single channel and the plurality of images.
21 . The method of claim 20 , wherein the generating of the reflectance map comprises applying an element-wise division to the input image using the illumination map and the illuminant color vector.
22 . The method of claim 20 , wherein the illumination extraction model comprises an encoder portion and a decoder portion, and the color map generated for each color channel from the input image is output from a convolutional layer of the encoder portion.