Reducing effects of light diffraction in under display camera (UDC) systems
A method includes obtaining, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern. The first diffraction pattern and the second diffraction pattern are related through a transformation. The method also includes generating a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames. The method further includes combining the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.
1 . A method comprising:
obtaining, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern related through a transformation;
generating a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames; and
combining the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.
2 . The method of claim 1 , wherein combining the first and second deblurred images comprises:
performing an alignment operation to generate aligned versions of the first and second deblurred images, the alignment operation exploiting the complementary types of image artifacts created by the first and second diffraction patterns; and
merging the aligned versions of the first and second deblurred images.
3 . The method of claim 2 , wherein performing the alignment operation comprises at least one of:
performing a global registration operation in order to globally align the first deblurred image and the second deblurred image; and
performing a local alignment operation in order to locally warp at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.
4 . The method of claim 3 , wherein at least one of:
the global registration operation comprises a feature extraction operation and a feature matching operation; and
the local alignment operation comprises a non-rigid warping operation.
5 . The method of claim 2 , wherein merging the aligned versions of the first and second deblurred images comprises:
merging the aligned versions of the first and second deblurred images using at least one merging operation that is selected based on an image quality criterion, the image quality criterion able to vary within different portions of the first and second deblurred images.
6 . The method of claim 5 , wherein the at least one merging operation comprises at least one of:
an averaging operation that is selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio;
a weighted averaging operation that is selected when the image quality criterion specifies obtaining a lower distortion, wherein weighting in the weighted averaging operation is based on local contrast; and
a minimum operation that is selected when the image quality criterion specifies removing diffraction artifacts.
7 . The method of claim 1 , wherein:
the first diffraction pattern is associated with a first pattern of a display under which the at least one under display camera is positioned;
the second diffraction pattern is associated with a second pattern of the display; and
the second diffraction pattern is obtained by at least one of:
rotating the at least one under display camera;
rotating at least a portion of the display relative to the at least one under display camera;
integrating different grid patterns into different portions of the display; and
rotating an electronic device that includes the display and the at least one under display camera.
8 . An electronic device comprising:
a display;
at least one under display camera positioned under the display; and
at least one processor configured to:
obtain, using the at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern related through a transformation;
generate a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames; and
combine the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.
9 . The electronic device of claim 8 , wherein, to combine the first and second deblurred images, the at least one processor is configured to:
perform an alignment operation to generate aligned versions of the first and second deblurred images, the alignment operation exploiting the complementary types of image artifacts created by the first and second diffraction patterns; and
merge the aligned versions of the first and second deblurred images.
10 . The electronic device of claim 9 , wherein, to perform the alignment operation, the at least one processor is configured to at least one of:
perform a global registration operation in order to globally align the first deblurred image and the second deblurred image; and
perform a local alignment operation in order to locally warp at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.
11 . The electronic device of claim 10 , wherein at least one of:
the global registration operation comprises a feature extraction operation and a feature matching operation; and
the local alignment operation comprises a non-rigid warping operation.
12 . The electronic device of claim 9 , wherein, to merge the aligned versions of the first and second deblurred images, the at least one processor is configured to merge the aligned versions of the first and second deblurred images using at least one merging operation that is selected based on an image quality criterion, the image quality criterion able to vary within different portions of the first and second deblurred images.
13 . The electronic device of claim 12 , wherein the at least one merging operation comprises at least one of:
an averaging operation that is selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio;
a weighted averaging operation that is selected when the image quality criterion specifies obtaining a lower distortion, wherein weighting in the weighted averaging operation is based on local contrast; and
a minimum operation that is selected when the image quality criterion specifies removing diffraction artifacts.
14 . The electronic device of claim 8 , wherein:
the first diffraction pattern is associated with a first pattern of the display;
the second diffraction pattern is associated with a second pattern of the display; and
the second diffraction pattern is obtained by at least one of:
rotation of the at least one under display camera;
rotation of at least a portion of the display relative to the at least one under display camera;
integration of different grid patterns into different portions of the display; and
rotation of the electronic device.
15 . A non-transitory machine readable medium containing instructions that when executed cause at least one processor of an electronic device to:
obtain, using at least one under display camera, one or more first image frames associated with a first diffraction pattern and one or more second image frames associated with a second diffraction pattern, the first diffraction pattern and the second diffraction pattern related through a transformation;
generate a first deblurred image using the one or more first image frames and a second deblurred image using the one or more second image frames; and
combine the first and second deblurred images while exploiting complementary types of image artifacts created by the first and second diffraction patterns to generate an image of a scene.
16 . The non-transitory machine readable medium of claim 15 , wherein the instructions that when executed cause the at least one processor to combine the first and second deblurred images comprise:
instructions that when executed cause the at least one processor to:
perform an alignment operation to generate aligned versions of the first and second deblurred images, the alignment operation exploiting the complementary types of image artifacts created by the first and second diffraction patterns; and
merge the aligned versions of the first and second deblurred images.
17 . The non-transitory machine readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to perform the alignment operation comprise:
instructions that when executed cause the at least one processor to at least one of:
perform a global registration operation in order to globally align the first deblurred image and the second deblurred image; and
perform a local alignment operation in order to locally warp at least a portion of the first deblurred image and locally align portions of the first deblurred image and portions of the second deblurred image.
18 . The non-transitory machine readable medium of claim 17 , wherein at least one of:
the global registration operation comprises a feature extraction operation and a feature matching operation; and
the local alignment operation comprises a non-rigid warping operation.
19 . The non-transitory machine readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to merge the aligned versions of the first and second deblurred images comprise:
instructions that when executed cause the at least one processor to merge the aligned versions of the first and second deblurred images using at least one merging operation that is selected based on an image quality criterion, the image quality criterion able to vary within different portions of the first and second deblurred images.
20 . The non-transitory machine readable medium of claim 19 , wherein the at least one merging operation comprises at least one of:
an averaging operation that is selected when the image quality criterion specifies obtaining a higher signal-to-noise ratio;
a weighted averaging operation that is selected when the image quality criterion specifies obtaining a lower distortion, wherein weighting in the weighted averaging operation is based on local contrast; and
a minimum operation that is selected when the image quality criterion specifies removing diffraction artifacts.