IP Library › Granted Patent US 12,216,277
Granted Patent B2
US 12,216,277 · App. 17/963,609 · Granted Feb 4, 2025

Optical element for deconvolution

Inventor: Ernest Rehmatulla Post (San Francisco, CA)
Assignee: Samsung Electronics Co., Ltd.
G02B27/0068G06T5/20G06T5/73H04N23/53G06T2207/20021G06T2207/20212
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,216,277
App. No.
17/963,609
Granted
Feb 4, 2025
Kind
B2
Abstract

In one embodiment, a method of constructing a corrective phase mask for an optical element, includes propagating, for each of one or more wavelengths, a point source field from an object plane to a corrective mask plane to determine a source field and propagating, for each of the one or more wavelengths, the point source field from an image plane to the corrective mask plane to determine an image field. The method may further include determining, for each of the one or more wavelengths, a phase modulation field based on the source field and the image field; and determining a multi-wavelength phase modulation field based on combining the phase modulation field for each of the one or more wavelengths.

Claims (42)

1. A method of constructing a corrective phase mask for a physical corrective optical element, comprising:

propagating, for each of one or more wavelengths, a point source field from an object plane to a corrective mask plane to determine a source field;

propagating, for each of the one or more wavelengths, the point source field from an image plane to the corrective mask plane to determine an image field;

determining, for each of the one or more wavelengths, a phase modulation field based on the source field and the image field;

determining a multi-wavelength phase modulation field based on combining the phase modulation field for each of the one or more wavelengths; and

constructing the physical corrective optical element based on the determined multi-wavelength phase modulation field to deblur distorted images without having to deconvolve the image computationally.

2. The method of claim 1 , wherein the modulation field for each of the one or more wavelengths is combined based on at least one of: super-positioning or random sampling.

3. The method of claim 1 , further comprising providing an optical element based on the multi-wavelength phase modulation field.

4. The method of claim 3 , wherein the optical element is disposed in a computing device comprising:

a display;

an image sensor disposed behind the display; and

the optical element disposed in front of the image sensor, wherein the optical element deblurs optical blurring caused by the display.

5. The method of claim 4 , wherein the optical element is disposed behind the display.

6. The method of claim 4 , wherein the optical element is disposed spatially coincident with the display.

7. The method of claim 1 , wherein for each of the one or more wavelengths the phase modulation field comprises the source field for that wavelength divided by the image field for that wavelength.

8. The method of claim 1 , wherein the point source field from the object plane to the corrective mask plane and the point source field from the image plane to the corrective mask plane each comprise incoherent light.

9. A system comprising:

a physical corrective optical element made by a process comprising:

propagating, for each of one or more wavelengths, a point source field from an object plane to a corrective mask plane to determine a source field;

propagating, for each of the one or more wavelengths, the point source field from an image plane to the corrective mask plane to determine an image field;

determining, for each of the one or more wavelengths, a phase modulation field based on the source field and the image field;

determining a multi-wavelength phase modulation field based on combining the phase modulation field for each of the one or more wavelengths; and

fabricating the physical corrective optical element based on the multi-wavelength phase modulation field to deblur distorted images without having to deconvolve the image computationally.

10. The system of claim 9 , wherein the modulation field for each of the one or more wavelengths is combined based on at least one of: super-positioning or random sampling.

11. The system of claim 9 , further comprising:

a display; and

an image sensor disposed behind the display, wherein the optical element is disposed in front of the image sensor and the optical element deblurs optical blurring caused by the display.

12. The system of claim 11 , wherein the optical element is disposed behind the display.

13. The system of claim 11 , wherein the optical element is disposed spatially coincident with the display.

14. The system of claim 9 , wherein for each of the one or more wavelengths the phase modulation field comprises the source field for that wavelength divided by the image field for that wavelength.

15. The system of claim 9 , wherein the point source field from the object plane to the corrective mask plane and the point source field from the image plane to the corrective mask plane each comprise incoherent light.

16. A method comprising:

receiving, at an optical element of a device, light from an object external to the device;

modulating, by the optical element, the light from the object, wherein the modulation by the optical element deblurs blurring caused by one or more components of the device without deconvolving the light from image computationally; and

capturing, by an image sensor, the modulated light.

17. The method of claim 16 , wherein the device comprises a computing device comprising:

a display;

the image sensor disposed behind the display; and

the optical element disposed in front of the image sensor, wherein the optical element deblurs optical blurring caused by the display.

18. The method of claim 17 , wherein the optical element is disposed behind the display.

19. The method of claim 17 , wherein the optical element is disposed spatially coincident with the display.

20. The method of claim 16 , further comprising displaying, on a display of the device, an image of the object as captured by the image sensor.

Continuity (2)
Provisional Application 63255819 · Oct 14, 2021
Related Publication 20230119549A1 · Apr 20, 2023
References Cited (157)
US 3971065A · Bayer · 1976 [cited by applicant]
US 6454414B1 · Ting · 2002 [cited by applicant]
US 7053613B2 · Lin · 2006 [cited by applicant]
US 8041142B2 · Schafer · 2011 [cited by applicant]
US 8433152B2 · Watanabe · 2013 [cited by applicant]
US 8582911B2 · Kim · 2013 [cited by applicant]
US 8587703B2 · Lelescu · 2013 [cited by applicant]
US 9220481B2 · Park · 2015 [cited by applicant]
US 9338354B2 · Hong · 2016 [cited by applicant]
US 9582862B2 · Zhang · 2017 [cited by applicant]
US 9654707B2 · Oniki · 2017 [cited by applicant]
US 9911208B2 · Zhou · 2018 [cited by applicant]
US 9916656B2 · Choi · 2018 [cited by applicant]
US 9947901B2 · Shedletsky · 2018 [cited by applicant]
US 9948849B2 · Kim · 2018 [cited by applicant]
US 10032254B2 · Harmeling · 2018 [cited by applicant]
US 10062153B2 · Oniki · 2018 [cited by applicant]
US 10083335B2 · Zhang · 2018 [cited by applicant]
US 10151933B2 · Siddiqui · 2018 [cited by applicant]
US 10178381B2 · Hall · 2019 [cited by applicant]
US 10191577B2 · Choi · 2019 [cited by applicant]
US 10217190B2 · Liu · 2019 [cited by applicant]
US 10416087B2 · Zhang · 2019 [cited by applicant]
US 10595724B2 · Lai · 2020 [cited by applicant]
US 10642059B2 · Soskind · 2020 [cited by applicant]
US 10656437B2 · Limon · 2020 [cited by applicant]
US 11003088B2 · Sorg · 2021 [cited by applicant]
US 11038143B2 · Moon · 2021 [cited by applicant]
US 11073712B2 · Yeke Yazdandoost · 2021 [cited by applicant]
US 11272106B1 · Lee · 2022 [cited by applicant]
US 11575865B2 · Liu · 2023 [cited by applicant]
US 11721001B2 · Liu · 2023 [cited by examiner]
US 11792515B2 · Lee · 2023 [cited by applicant]
US 20030002746A1 · Kusaka · 2003 [cited by applicant]
US 20060103951A1 · Bell · 2006 [cited by applicant]
US 20060256226A1 · Alon · 2006 [cited by applicant]
US 20080013850A1 · Sakurai · 2008 [cited by applicant]
US 20080068660A1 · Lace · 2008 [cited by applicant]
US 20080165261A1 · Kamo · 2008 [cited by applicant]
US 20080166115A1 · Sachs · 2008 [cited by applicant]
US 20080218597A1 · Cho · 2008 [cited by applicant]
US 20080292135A1 · Schafer · 2008 [cited by applicant]
US 20090147111A1 · Litvinov · 2009 [cited by applicant]
US 20090263043A1 · Cristobal · 2009 [cited by applicant]
US 20100073518A1 · Yeh · 2010 [cited by applicant]
US 20100188528A1 · Iwata · 2010 [cited by applicant]
US 20110019056A1 · Hirsch · 2011 [cited by applicant]
US 20110075257A1 · Hua · 2011 [cited by applicant]
US 20110158541A1 · Watanabe · 2011 [cited by applicant]
US 20110221888A1 · Choi · 2011 [cited by applicant]
US 20110285680A1 · Nakamura · 2011 [cited by applicant]
US 20120057072A1 · Yamashita · 2012 [cited by applicant]
US 20120162490A1 · Chung · 2012 [cited by applicant]
US 20120327277A1 · Myhrvold · 2012 [cited by applicant]
US 20130010077A1 · Nguyen · 2013 [cited by applicant]
US 20130147778A1 · Ninan · 2013 [cited by applicant]
US 20130182062A1 · Son · 2013 [cited by applicant]
US 20130308007A1 · Tanaka · 2013 [cited by applicant]
US 20130321686A1 · Tan · 2013 [cited by applicant]
US 20130336597A1 · Maeda · 2013 [cited by applicant]
US 20140044314A1 · Sezer · 2014 [cited by applicant]
US 20150049165A1 · Choi · 2015 [cited by applicant]
US 20150101411A1 · Zalev · 2015 [cited by applicant]
US 20150207962A1 · Sugimoto · 2015 [cited by applicant]
US 20150338639A1 · Mtsumoto · 2015 [cited by examiner]
US 20160062100A1 · Cohen · 2016 [cited by examiner]
US 20160180510A1 · Grau · 2016 [cited by applicant]
US 20160248975A1 · Choi · 2016 [cited by applicant]
US 20160277658A1 · Kim · 2016 [cited by applicant]
US 20160371821A1 · Hayashi · 2016 [cited by applicant]
US 20170076430A1 · Xu · 2017 [cited by applicant]
US 20170104897A1 · Kang · 2017 [cited by applicant]
US 20170212613A1 · Hwang · 2017 [cited by applicant]
US 20170316552A1 · Hanocka · 2017 [cited by applicant]
US 20180038768A1 · Hofmann · 2018 [cited by applicant]
US 20180052050A1 · Menon · 2018 [cited by applicant]
US 20180116500A1 · Escalier · 2018 [cited by applicant]
US 20180129061A1 · Shinohara · 2018 [cited by applicant]
US 20180198980A1 · Takagi · 2018 [cited by applicant]
US 20180211420A1 · Yoo · 2018 [cited by applicant]
US 20190212544A1 · Heber · 2019 [cited by applicant]
US 20190213717A1 · Oniki · 2019 [cited by applicant]
US 20190327417A1 · Moriuchi · 2019 [cited by applicant]
US 20190355101A1 · Chen · 2019 [cited by applicant]
US 20200159102A1 · Kouyama · 2020 [cited by examiner]
US 20200166807A1 · Sasaki · 2020 [cited by applicant]
US 20200169725A1 · Hua · 2020 [cited by applicant]
US 20200209604A1 · Chen · 2020 [cited by applicant]
US 20200321561A1 · Park · 2020 [cited by applicant]
US 20200389575A1 · Gove · 2020 [cited by applicant]
US 20200394964A1 · Hyun · 2020 [cited by applicant]
US 20210029336A1 · Liu · 2021 [cited by applicant]
US 20210136335A1 · Tanaka · 2021 [cited by examiner]
US 20210152735A1 · Zhou · 2021 [cited by applicant]
US 20210193756A1 · Oh · 2021 [cited by applicant]
US 20210199952A1 · Cho · 2021 [cited by applicant]
US 20210210533A1 · Cho · 2021 [cited by applicant]
US 20210233976A1 · Lee · 2021 [cited by applicant]
US 20210302316A1 · Walter · 2021 [cited by applicant]
US 20220067889A1 · Kang · 2022 [cited by applicant]
US 20220086309A1 · Kim · 2022 [cited by applicant]
US 20220138924A1 · Kwon · 2022 [cited by applicant]
US 20220261966A1 · Liu · 2022 [cited by applicant]
US 20220277426A1 · Vyas · 2022 [cited by applicant]
US 20220292637A1 · Huang · 2022 [cited by applicant]
US 20230341264A1 · Houck · 2023 [cited by examiner]
CN 101414094A · 2009 [cited by applicant]
CN 112202991A · 2021 [cited by applicant]
CN 113053253A · 2021 [cited by applicant]
CN 1113067961A · 2021 [cited by applicant]
CN 108335268B · 2021 [cited by applicant]
CN 114331886A · 2022 [cited by applicant]
CN 110675347B · 2022 [cited by applicant]
JP 2008070566A · 2008 [cited by applicant]
JP 2019068378A · 2019 [cited by applicant]
JP 6652052B9 · 2020 [cited by applicant]
KR 101894391B1 · 2018 [cited by applicant]
KR 1020220014764A · 2022 [cited by applicant]
WO WO2010081229A1 · 2010 [cited by applicant]
WO WO2016154392A1 · 2016 [cited by applicant]
WO WO2017117152A1 · 2017 [cited by applicant]
WO WO2021122471A1 · 2021 [cited by applicant]
WO WO2022005157A1 · 2022 [cited by applicant]
Notice of Allowance in U.S. Appl. No. 17/380,995, Dec. 21, 2022. [cited by applicant]
PCT Search Report and Written Opinion in PCT/KR2022/015641, Jan. 26, 2023. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/KR2023/007313, Aug. 30, 2023. [cited by applicant]
Image Restoration for Under-Display Camera, Yuqian Zhou et al., CVPR 2021, accessed on Oct. 25, 2022 at https://openaccess.thecvf.com/CVPR2021?day=all. [cited by applicant]
Notice of Allowance in U.S. Appl. No. 17/176,535, Mar. 8, 2023. [cited by applicant]
Anqi Yang et al., 'Designing Display Pixel Layouts for Under-Panel Cameras', IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, No. 7, pp. 2245-2256, Apr. 27, 2021. [cited by applicant]
PCT Search Report and Written Decision in PCT/KR2022/011598, Nov. 16, 2022. [cited by applicant]
Final Office Action in U.S. Appl. No. 17/176,535, Nov. 7, 2022. [cited by applicant]
Levin, A. et al., “Image and Depth from a Conventional Camera with a Coded Aperture,” ACM Transactions on Graphics, vol. 26, No. 3, Article 70, Publication date Jul. 2007, DOI 10.1145/1239451.123952, http://doi.acm.org/… [cited by applicant]
Hong, J., et al., “Three-dimensional display technologies of recent interest: principles, status, and issues [Invited],” (Doc. ID 152226), Applied Optics, vol. 50, No. 34, , Dec. 1, 2011, https://www.researchgate.net/pu… [cited by applicant]
Ren, Ng, “Digital light field photography,” PhD dissertation, Stanford University, Jul. 2006, 203 pgs. [cited by applicant]
Qin, Zong, et al., “See-Through Image Blurring of Transparent Organic Light-Emitting Diodes Display: Calculation Method Based on Diffraction and Analysis of Pixel Structures,” Journal of Display Technology, vol. 12, No.… [cited by applicant]
Richardson, William Hadley, “Bayesian-Based Iterative Method of Image Restoration,” Journal of Optical Society of America, vol. 62, No. 1, Jan. 1972, pp. 55-59 (5 pgs). [cited by applicant]
Lucy, L. B., “An Iterative Technique for the Rectification of Observed Distributions,” The Astronomical Journal, vol. 79, No. 6, Jun. 1974, © American Astronomical Society, provided by the NASA Astrophysics Data System,… [cited by applicant]
Heide, Felix, et al., “ProxImaL: Efficient Image Optimization Using Proximal Algorithms,” SIGGRAPH 16 Technical paper, Jul. 24-28, 2016, Anaheim, CA. SIGGRAPH '16 Technical Paper, Jul. 24-28, 2016, Anaheim, CA, ISBN: 97… [cited by applicant]
Sitzmann, Vincent., et al., “End-to-End Optimization of Optics and Image Processing for Achromatic Extended Depth of Field and Super-Resolution Imaging,” © 2018 ACM 0730-0301/2018/8-ART114, https://doi.org/10.1145/31975… [cited by applicant]
Lai, Richard, “Oppo's under-screen camera is real and taking photos in Shanghai,” https://www.engadget.com/2019-06-26-oppo-under-screen-camera-mwc-shanghai.html, Jun. 26, 2019, 8 pgs. [cited by applicant]
International Search Report and Written Opinion for International App. No. PCT/KR2020/009807, Oct. 26, 2020. [cited by applicant]
European Search Report in EP 20846484.2, Mar. 14, 2022. [cited by applicant]
Non-Final Office Action in U.S. Appl. No. 16/935,946, Apr. 5, 2022. [cited by applicant]
Notice of Allowance in U.S. Appl. No. 16/935,946, Jul. 6, 2022. [cited by applicant]
PCT Search Report and Written Decision in PCT/KR2022/001920, May 13, 2022. [cited by applicant]
Non-Final Office Action in U.S. Appl. No. 17/176,535, Mar. 7, 2022. [cited by applicant]
Yang, Hang, Zhongbo Zhang, and Yujing Guan. “An adaptive parameter estimation for guided filter based image deconvolution.” [cited by applicant]
Youmaran, R., and A. Adler. “Combining regularization frameworks for image deblurring: optimization of combined hyper-parameters.” In [cited by applicant]
Non-Final Office Action in U.S. Appl. No. 17/380,995, Jun. 22, 2022. [cited by applicant]
PCT Search Report and written decision in PCT/KR2022/001024, May 10, 2022. [cited by applicant]
Notice of Allowance in U.S. Appl. No. 16/935,946, Sep. 30, 2022. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/KR2023/017304, Feb. 2, 2024. [cited by applicant]
Yangjie Wei et al., Blurring kernel extraction and super-resolution image reconstruction based on style generative adersarial networks, Optics Express vol. 29, Issue 26, Dec. 16, 2021. [cited by applicant]
Eric Yang, Variable Synthetic Depth of Field with Mobile Stereo Cameras, Computer Science Engineering, https://stanford.edu/class/ee367/sections 3.2, 4.3; 2020, retrieved on Jan. 9, 2024. [cited by applicant]
Soldevila F et al: “Phase imaging by spatial wavefront sampling”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Nov. 11, 2017 (Nov. 11, 2017), XP081287443, DOI: 10.1364/OPT… [cited by applicant]
Katkovnik Vladhviir et al: “A novel binary and multilevel phase masks for enhanced depth-of-focus infrared imaging”, 2018 52nd Asilomar Conference on Signals, Systems, and Computers, IEEE, Oct. 28, 2018 (Oct. 28, 2018),… [cited by applicant]
European Patent Office Extended European Search Report in Application No. 228814190-1207/4327058 PCT/KR2022015641, Sep. 19, 2024. [cited by applicant]