Apparatus and method for mapping raw images between different camera sensors under arbitrary illuminations

1 . An electronic device for processing image data, the electronic device comprising:

at least one memory configured to store instructions; and

at least one processor configured to execute the instructions to:

obtain a raw input image that is captured using a first image sensor and an input illumination condition corresponding to the raw input image;

estimate, based on the input illumination condition, a color transform that maps a characteristic of the first image sensor to a characteristic of a second image sensor; and

generate a raw output image having the characteristic of the second image sensor based on the raw input image and the color transform.

2 . The electronic device of claim 1 , wherein the characteristic of the first image sensor comprises a color response of the first image sensor which corresponds to a spectral sensitivity of the first image sensor, and

wherein the characteristic of the second image sensor comprises a color response of the second image sensor which corresponds to a spectral sensitivity of the second image sensor.

3 . The electronic device of claim 1 , wherein the at least one processor is further configured to execute the instructions to estimate the color transform using an artificial intelligence (AI) model which is trained based on a plurality of raw image pairs corresponding to a plurality of illumination conditions, and

wherein each raw image pair comprises a first raw image captured using the first image sensor under an illumination condition from among the plurality of illumination conditions, and a second raw image captured using the second image sensor under the illumination condition.

4 . The electronic device of claim 3 , wherein the plurality of raw image pairs are obtained using a variable light source configured to apply the plurality of illumination conditions to an object.

5 . The electronic device of claim 4 , wherein the object comprises a color calibration pattern including a plurality of colors, and

wherein the color calibration pattern comprises a plurality of textures corresponding to the plurality of colors.

6 . The electronic device of claim 1 , wherein the raw input image and the raw output image are not white-balance corrected.

7 . The electronic device of claim 1 , further comprising an input interface,

wherein the at least one processor is further configured to obtain the color transform and generate the raw output image based on receiving information about the second image sensor through the input interface.

8 . The electronic device of claim 1 , wherein the first image sensor is included in an external device, and the second image sensor is included in the electronic device,

wherein the electronic device further comprises a communication interface configured to receive the raw input image from the external device, and

wherein the at least one processor is further configured to execute the instructions to convert the raw input image that is received from the external device, into the raw output image having the characteristic of the second image sensor based on the color transform.

9 . The electronic device of claim 1 , wherein the at least one processor is further configured to execute the instructions to:

obtain an image data set comprising a plurality of raw first images captured using the first image sensor;

obtain, using the transform estimator, a plurality of color transforms that map the characteristic of the first image sensor to the characteristic of the second image sensor, wherein the plurality of color transforms comprises the color transform;

create a transformed data set comprising a plurality of raw second images having the characteristic of the second image sensor based on the plurality of raw first images and the plurality of color transforms, wherein the plurality of raw second images comprises the raw output image; and

input the transformed data set to an artificial intelligence (AI)-based image processing model to train the AI-based image processing model.

10 . The electronic device of claim 1 , further comprising the first image sensor and the second image sensor,

wherein the raw input image comprises a first raw input video frame,

wherein the raw output image comprises a raw output video frame, and

wherein the at least one processor is further configured to:

obtain a second raw input video frame that is captured using the second image sensor; and

generate a video based on the raw output video frame and the second raw input video frame.

11 . The electronic device of claim 1 , wherein, to estimate the color transform, the at least one processor is further configured to execute the instructions to:

provide the input illumination condition as an input to an artificial intelligence (AI) model; and

obtain the color transform based on an output of the AI model.

12 . A method for processing image data, the method being performed by at least one processor and comprising:

obtaining a raw input image that is captured using a first image sensor and an input illumination condition corresponding to the raw input image;

estimating, based on the input illumination condition, a color transform that maps a characteristic of the first image sensor to a characteristic of a second image sensor; and

generating a raw output image having the characteristic of the second image sensor based on the raw input image and the color transform.

13 . The method of claim 12 , wherein the characteristic of the first image sensor comprises a color response of the first image sensor which corresponds to a spectral sensitivity of the first image sensor, and

wherein the characteristic of the second image sensor comprises a color response of the second image sensor which corresponds to a spectral sensitivity of the second image sensor.

14 . The method of claim 12 , wherein the estimating is performed using an artificial intelligence (AI) model which is trained based on a plurality of raw image pairs corresponding to a plurality of illumination conditions, and

wherein each raw image pair comprises a first raw image captured using the first image sensor under an illumination condition from among the plurality of illumination conditions, and a second raw image captured using the second image sensor under the illumination condition.

15 . The method of claim 14 , wherein the plurality of raw image pairs are obtained using a variable light source configured to apply the plurality of illumination conditions to an object.

16 . The method of claim 15 , wherein the object comprises a color calibration pattern including a plurality of colors, and

wherein the color calibration pattern comprises a plurality of textures corresponding to the plurality of colors.

17 . The method of claim 12 , wherein the color transform is obtained and the raw output image is generated based on receiving information about the second image sensor through an input interface.

18 . The method of claim 12 , wherein the first image sensor is included in a first electronic device, and the second image sensor is included in a second electronic device which includes the at least one processor,

wherein the second electronic device comprises a communication interface configured to receive the raw input image from the first electronic device, and

wherein the method further comprises converting the raw input image that is received from the first electronic device, into the raw output image having the characteristic of the second image sensor based on the color transform.

19 . The method of claim 12 , wherein the method further comprises:

obtaining an image data set comprising a plurality of raw first images captured using the first image sensor;

estimating a plurality of color transforms that map the characteristic of the first image sensor to the characteristic of the second image sensor, wherein the plurality of color transforms comprises the color transform;

creating a transformed data set comprising a plurality of raw second images having the characteristic of the second image sensor based on the plurality of raw first images and the plurality of color transforms, wherein the plurality of raw second images comprises the raw output image; and

inputting the transformed data set to an artificial intelligence (AI)-based image processing model to train the AI-based image processing model.

20 . The method of claim 12 , wherein the at least one processor, the first image sensor, and the second image sensor are included in an electronic device,

wherein the raw input image comprises a first raw input video frame,

wherein the raw output image comprises a raw output video frame, and

wherein the method further comprises:

obtaining a second raw input video frame that is captured using the second image sensor; and

generating a video based on the raw output video frame and the second raw input video frame.

21 . A non-transitory computer-readable medium configured to store instructions which, when executed by at least one processor of a device for processing image data, cause the at least one processor to:

obtain a raw input image that is captured using a first image sensor and an input illumination condition corresponding to the raw input image;

estimate, based on the input illumination condition, a color transform that maps a characteristic of the first image sensor to a characteristic of a second image sensor; and

generate a raw output image having the characteristic of the second image sensor based on the raw input image and the color transform.