Electronic device that compensates for light passing through a display, and driving method for the same

1. An electronic device comprising:

a display panel comprising a display region to display an image;

a camera module under the display panel, overlapping the display region, and configured to output an image signal; and

a compensator configured to generate a compensated image signal by compensating the image signal through a compensation algorithm,

wherein the compensation algorithm is trained by using a target image and a simulation image that is generated to reproduce an image captured based on light passing through the display panel,

wherein the target image is a composite image synthesized based on a first processed image, for restoration,

wherein the simulation image is generated by compositing a second processed image either with an artificial light source or with a first measured image captured based on the light passing through the display panel,

wherein each of the first processed image and the second processed image is obtained by processing a first image, and

wherein the first image comprises an image generated through light that does not pass through the display panel.

2. A method for driving an electronic device, the method comprising:

receiving light passing through a display panel;

generating an image signal based on the light;

compensating the image signal with a compensation algorithm to generate a compensated image signal, wherein the compensation algorithm is trained with training data comprising a first comparison image and a second comparison image; and

displaying, on the display panel, a compensated image based on the compensated image signal,

wherein the first comparison image is a composite image synthesized based on a first processed image, for restoration,

wherein the second comparison image comprises a simulation image reproducing an image that is generated by the light passing through the display panel, the simulation image being generated by compositing a second processed image formed based on light that does not pass through the display panel either with an artificial light source or with a first measured image captured based on the light passing through the display panel, and

wherein each of the first processed image and the second processed image is obtained by processing a first image.

3. The method of claim 2 , wherein the training data is generated by:

obtaining the first image;

generating a second image by performing pre-processing with respect to the first image;

generating a third image by adding the artificial light source to the second image; and

generating the first comparison image by performing a clipping process and a gamma process with respect to the third image.

4. The method of claim 3 , wherein the training data is generated by:

generating a fourth image by performing a space-variant convolution operation with respect to the third image;

generating a fifth image by applying a characteristic to the fourth image; and

generating the second comparison image by performing the clipping process and the gamma process with respect to the fifth image.

5. The method of claim 4 , wherein the first image comprises an image generated through light that does not pass through the display panel.

6. The method of claim 2 , wherein the training data is generated by:

obtaining the first image;

generating a second image by performing pre-processing with respect to the first image; and

generating the first comparison image by compositing the second image with a target light source image.

7. The method of claim 6 , wherein the training data is generated by:

generating a third image by performing a space-variant convolution operation with respect to the second image; and

generating the second comparison image by compositing the third image with the first measured image.

8. The method of claim 7 , wherein the target light source image is generated by:

obtaining a second measured image by binarizing the first measured image;

estimating a position, a size, and a shape of a light source of the second measured image;

generating a target light source by using a profile; and

generating the target light source image based on the target light source.

9. The method of claim 8 , wherein the first image comprises an image generated by the light that does not pass through the display panel.

10. The method of claim 2 , wherein the compensation algorithm is trained by a training method comprising:

generating a first high dynamic range (HDR) image by restoring the simulation image through an HDR image restoration neural network; and

comparing the first HDR image with the first comparison image.

11. The method of claim 10 , wherein the training method further comprises:

generating a second HDR image through deconvolution for the first HDR image using a point spread function;

generating a restored image by performing a first clipping process and a first gamma process with respect to the second HDR image; and

comparing the restored image with a target restoration image,

wherein the target restoration image is generated by performing a second clipping process and a second gamma process with respect to the first comparison image.

12. The method of claim 11 , wherein the training data is generated by:

obtaining the first image;

generating a second image by performing pre-processing with respect to the first image; and

generating a third image by adding the artificial light source to the second image,

wherein the first comparison image comprises a target HDR image.

13. The method of claim 12 , wherein the training data is further generated by:

generating a fourth image by performing a space-variant convolution operation with respect to the third image;

generating a fifth image by applying another characteristic to the fourth image; and

generating the second comparison image by performing a clipping process and a gamma process with respect to the fifth image.

14. The method of claim 2 , wherein the compensation algorithm is trained by a training method further comprising:

generating a plurality of comparison images by adjusting exposure of the second comparison image; and

generating a restored image by providing the plurality of comparison images to an image restoration neural network.

15. The method of claim 14 , wherein the training data is generated by:

obtaining the first image;

generating a second image by performing pre-processing with respect to the first image;

generating a third image by adding the artificial light source to the second image;

generating a fourth image by performing a space-variant convolution operation with respect to the third image; and

generating the second comparison image by applying other characteristics to the fourth image, wherein the second comparison image comprises a composite HDR image.

16. The method of claim 2 , wherein the compensation algorithm is trained by a training method comprising:

generating a first low-resolution image by down-sampling the second comparison image;

generating a second low-resolution image by providing the first low-resolution image to a low-resolution neural network;

generating a low-resolution guided filter based on the first low-resolution image and the second low-resolution image;

generating a guided filter by up-sampling the low-resolution guided filter; and

generating a restored image obtained by restoring the second comparison image using the guided filter.

17. The method of claim 16 , wherein the training method further comprises:

generating a high-resolution image by providing the second comparison image to a high-resolution pre-processing neural network;

calculating the high-resolution image using the guided filter and providing calculated data to a fusion neural network; and

generating the restored image using the fusion neural network.

18. The method of claim 17 , wherein the training method further comprises:

comparing the restored image with the first comparison image.

19. An electronic device comprising:

a display panel comprising a first display region having a first light transmittance, and a second display region having a second light transmittance that is greater than the first light transmittance;

a camera module under the display panel and overlapping the second display region, and configured to output an image signal; and

a compensator configured to generate a compensated image signal by compensating the image signal by a compensation algorithm, wherein the compensation algorithm is trained with training data comprising a first comparison image and a second comparison image,

wherein the first comparison image is a composite image synthesized based on a first processed image, for restoration,

wherein the second comparison image comprises a simulation image reproducing an image that is generated by light passing through the display panel, the simulation image being generated by compositing a second processed image formed based on light that does not pass through the display panel either with an artificial light source or with a first measured image captured based on the light passing through the display panel, and

wherein each of the first processed image and the second processed image is obtained by processing a first image.

20. The electronic device of claim 19 , wherein the second display region and the camera module are each provided in plurality,

wherein the plurality of camera modules overlap the plurality of second display regions in one-to-one correspondence, and

wherein image signals output from the plurality of camera modules are compensated by the compensation algorithm.