IP Library › Granted Patent US 12,682,475
Granted Patent B2
US 12,682,475 · App. 18/596,251 · Granted Jul 14, 2026

Large depth-of-field microscopic structured-light 3D imaging

Inventors: Song Zhang (West Lafayette, IN); Liming Chen (West Lafayette, IN)
Assignee: PURDUE RESEARCH FOUNDATION
G06T7/571G03B17/54G06T17/00H04N23/12H04N23/743G06T2207/10016
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,682,475
App. No.
18/596,251
Filed
Mar 5, 2024
Granted
Jul 14, 2026
Kind
B2
Examiner
DANG, PHILIP
Art Unit
2488
USPC
348/46
Abstract

Systems and methods of large depth-of-field microscopic structured-light 3D imaging include and/or implement capturing a plurality of images of a scene using a plurality of focus settings, each of the plurality of images corresponding to one of the plurality of focus settings, the plurality of images corresponding to a focal stack; processing the plurality of images; merging the plurality of images into an all-in-focus phase map; and reconstructing a 3D shape of the scene based on the all-in-focus phase map.

Claims (53)

1 . A method of capturing a three-dimensional (3D) image, the method comprising:

capturing a plurality of images of a scene using a plurality of focus settings, each of the plurality of images corresponding to one of the plurality of focus settings, the plurality of images corresponding to a focal stack, wherein the operation of capturing includes capturing, by a camera, a plurality of fringe images with different frequencies;

processing the plurality of images, wherein the operation of processing includes:

computing a high-frequency phase map and a lower-frequency phase map for respective ones of the plurality of fringe images,

estimating, using the lower-frequency phase map, affine warps that minimize a phase difference between different focus settings of the camera,

applying the affine warps to the high-frequency phase map to generate an aligned fringe contrast map, and

measuring, using the aligned fringe contrast map, a focus level for a plurality of pixels of an image sensor of the camera;

merging the plurality of images into an all-in-focus phase map; and

reconstructing a 3D shape of the scene based on the all-in-focus phase map.

2 . The method of claim 1 , wherein the operation of capturing includes changing a focal length of the camera to capture multi-focus images.

3 . The method of claim 2 , wherein the camera includes an electrically tunable lens (ETL) that allows for capturing the multi-focus images without mechanical motion.

4 . The method of claim 1 , wherein the operation of merging includes stitching the high-frequency phase maps for each of the plurality of images into the all-in-focus phase map using the aligned fringe contrast map based on the measured focus levels.

5 . The method of claim 1 , further comprising determining a plurality of effective focal planes, wherein in the operation of capturing, the plurality of focus settings correspond to the plurality of effective focal planes.

6 . The method of claim 5 , further comprising capturing a focal sweep image, including changing a focal length of the camera during a camera exposure process.

7 . The method of claim 6 , further comprising generating an initial 3D point cloud based on the focal sweep image.

8 . The method of claim 7 , wherein the operation of determining includes:

calculating a maximum depth value of the scene based on the initial 3D point cloud;

calculating a minimum depth value of the scene based on the initial 3D point cloud;

generating a plurality of preliminary focal plane positions based on the maximum depth value and the minimum depth value; and

for each of the plurality of preliminary focal plane positions:

comparing an image parameter of the initial 3D point cloud to a threshold, and

in response to a determination that the image parameter exceeds the threshold, marking the preliminary focal plane position as one of the plurality of effective focal planes.

9 . The method of claim 8 , wherein each of the plurality of focus settings corresponds to one of the plurality of effective focal planes.

10 . The method of claim 1 , wherein the operation of capturing the plurality of fringe images includes capturing, by the camera, a plurality of vertical fringe images with different frequencies and a plurality of horizontal fringe images with different frequencies.

11 . An imaging system, comprising:

a projector configured to emit a projection light;

a first optical system configured to direct the projection light from the projector to a stage on which an imaging target is located;

an image sensor configured to receive an imaging light;

a second optical system configured to direct the imaging light from the stage to the image sensor; and

a controller configured to cause the system to perform operations comprising:

capturing a plurality of images of the imaging target using a plurality of focus settings, each of the plurality of images corresponding to one of the plurality of focus settings, the plurality of images corresponding to a focal stack, wherein the operation of capturing includes capturing, by the image sensor, a plurality of fringe images with different frequencies,

processing the plurality of images, wherein the operation of processing includes:

computing a high-frequency phase map and a lower-frequency phase map for respective ones of the plurality of fringe images,

estimating, using the lower-frequency phase map, affine warps that minimize a phase difference between different focus settings of the image sensor,

applying the affine warps to the high-frequency phase map to generate an aligned fringe contrast map, and

measuring, using the aligned fringe contrast map, a focus level for a plurality of pixels of the image sensor,

merging the plurality of images into an all-in-focus phase map, and

reconstructing a three-dimensional (3D) shape of the imaging target based on the all-in-focus phase map.

12 . The system of claim 11 , wherein the controller includes a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the system to perform the operations.

13 . The system of claim 11 , wherein the second optical system includes an electrically tunable lens (ETL) configured to capture multi-focus images without mechanical motion.

14 . The system of claim 11 , wherein the operation of capturing includes changing a focal length of the image sensor to capture multi-focus images.

15 . The system of claim 11 , wherein the operation of merging includes stitching the high-frequency phase maps for each of the plurality of images into the all-in-focus phase map using the aligned fringe contrast map based on the measured focus levels.

16 . The system of claim 11 , the operations further comprising capturing a focal sweep image, including changing a focal length of the image sensor during an image sensor exposure process.

17 . The system of claim 16 , the operations further comprising generating an initial 3D point cloud based on the focal sweep image.

18 . The system of claim 17 , further comprising determining a plurality of effective focal planes, wherein in the operation of capturing, the plurality of focus settings correspond to the plurality of effective focal planes, wherein the operation of determining includes:

calculating a maximum depth value of the imaging target based on the initial 3D point cloud;

calculating a minimum depth value of the imaging target based on the initial 3D point cloud;

generating a plurality of preliminary focal plane positions based on the maximum depth value and the minimum depth value; and

for each of the plurality of preliminary focal plane positions:

comparing an image parameter of the initial 3D point cloud to a threshold, and

in response to a determination that the image parameter exceeds the threshold, marking the preliminary focal plane position as one of the plurality of effective focal planes.

19 . The system of claim 18 , wherein each of the plurality of focus settings corresponds to one of the plurality of effective focal planes.

20 . The system of claim 11 , wherein the operation of capturing the plurality of fringe images includes capturing, by the image sensor, a plurality of vertical fringe images with different frequencies and a plurality of horizontal fringe images with different frequencies.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE CORRECT ORIGINAL SUBMITTED ASSIGNMENT HAD OMITTED PAGE 6 ACCEPTED BY ASSIGNEE SIGNATURE PAGE PREVIOUSLY RECORDED AT REEL: 68432 FRAME: 292. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Sep 12, 2024
From: ZHANG, SONG; CHEN, LIMING
To: PURDUE RESEARCH FOUNDATION
Reel/Frame 068948/0750 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 28, 2024
From: ZHANG, SONG; CHEN, LIMING
To: PURDUE RESEARCH FOUNDATION
Reel/Frame 068432/0292 →
Continuity (2)
Provisional Application 63450047 · Mar 5, 2023
Related Publication 20240296577A1 · Sep 5, 2024
References Cited (59)
US 9191578B2 · Blayvas · 2015 [cited by examiner]
US 9292926B1 · Tang · 2016 [cited by examiner]
US 10948567B2 · Eberspach · 2021 [cited by examiner]
US 10955936B2 · Send · 2021 [cited by examiner]
US 11860292B2 · Eberspach · 2024 [cited by examiner]
US 20230341328A1 · Weingarten · 2023 [cited by examiner]
EP 4134644A1 · 2023 [cited by examiner]
Agarwala A, Dontcheva M, Agrawala M, Drucker S, Colburn A, Curless B, et al. Interactive digital photomontage. ACM Trans Graph 2004;23:294-302. [cited by applicant]
Baker S, Matthews I. Equivalence and efficiency of image alignment algorithms. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2001, p. I-1090-I-1097. [cited by applicant]
Boykov Y, Kolmogorov V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 2004;26:1124-37. [cited by applicant]
Boykov Y, Veksler O, Zabih R. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001;23:1222-39. [cited by applicant]
Brown LG. A survey of image registration techniques. ACM Comput Surv 1992;24:325-76. [cited by applicant]
Burt PJ, Kolczynski RJ. Enhanced image capture through fusion. 1993 (4th) International Conference on Computer Vision, 1993, p. 173-82. [cited by applicant]
Chen et al., “Calibration method for a multi-focus microscopic 3D imaging system,” Opt. Lett. 48, 4348-4351 (2023). [cited by applicant]
Chen et al., “Electrically tunable lens assisted absolute phase unwrapping for large depth-of-field 3D microscopic structured light imaging,” Opt. Lasers Eng. 174, 107967 (2024). [cited by applicant]
Chen et al., “Large depth-of-field microscopic structured light 3D imaging with focus stacking,” Opt. Lasers Eng. 167, 107623 (2023). [cited by applicant]
Chen L, Hu X, Zhang S. Calibration method for an extended depth-of-field microscopic structured light system. Opt Express 2022;30:166-78. [cited by applicant]
Clark D, Brown B. A Rapid Image Acquisition Method for Focus Stacking in Microscopy. Microscopy Today 2015;23:18-25. [cited by applicant]
Crete et al., “The blur effect: perception and estimation with a new no reference perceptual blur metric,” in Proc. SPIE 6492, Human Vision and Electronic Imaging XII, 649201 (2007). [cited by applicant]
Fischler MA, Bolles RC. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 1981;24:381-95. [cited by applicant]
Gupta M, Agrawal A, Veeraraghavan A, Narasimhan SG. Structured light 3D scanning in the presence of global illumination. CVPR 2011, 2011, p. 713-20. [cited by applicant]
Gupta M, Nayar SK. Micro Phase Shifting. 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, p. 813-20. [cited by applicant]
Hartley R, Zisserman A. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge: Cambridge Univ. Press; 2004. [cited by applicant]
Hasinoff SW, Kutulakos KN, Durand F, Freeman WT. Time-constrained photography. 2009 IEEE 12th International Conference on Computer Vision, 2009, p. 333-40. [cited by applicant]
Hasinoff SW, Kutulakos KN. Light-Efficient Photography. IEEE Transactions on Pattern Analysis and Machine Intelligence 2011;33:2203-14. [cited by applicant]
Hu et al., “Large depth-of-field three-dimensional shape measurement with the focal sweep technique” Opt. Express 28, 31197-31208 (2020). [cited by applicant]
Hu Y, Chen Q, Feng S, Tao T, Asundi A, Zuo C. A new microscopic telecentric stereo vision system—Calibration, rectification, and three-dimensional reconstruction. Opt Lasers Eng 2019;113:14-22. [cited by applicant]
Hu Y, Chen Q, Feng S, Tao T, Li H, Zuo C. Real-time microscopic 3D shape measurement based on optimized pulse-width-modulation binary fringe projection. Meas Sci Technol 2017;28:075010. [cited by applicant]
Hu Y, Chen Q, Feng S, Zuo C. Microscopic fringe projection profilometry: A review. Opt Lasers Eng 2020;135:106192. [cited by applicant]
Kolmogorov V, Zabin R. What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence 2004;26:147-59. [cited by applicant]
Korner et al., “One-grating projection for absolute three-dimensional profiling”, Opt. Eng. 40(8) 1653-1660 (Aug. 2001). [cited by applicant]
Kuthirummal et al., “Flexible Depth of Field Photography,” IEEE Trans. on Pattern Anal. Mach. Intell. 33, 58-71 (2011). [cited by applicant]
Li A, Peng X, Yin Y, Liu X, Zhao Q, Körner K, et al. Fringe projection based quantitative 3D microscopy. Optik 2013;124:5052-6. [cited by applicant]
Li B, Zhang S. Flexible calibration method for microscopic structured light system using telecentric lens. Opt Express 2015;23:25795-803. [cited by applicant]
Li B, Zhang S. Microscopic structured light 3D profilometry: Binary defocusing technique vs. sinusoidal fringe projection. Opt Lasers Eng 2017;96:117-23. [cited by applicant]
Li D, Liu C, Tian J. Telecentric 3D profilometry based on phase-shifting fringe projection. Opt Express 2014;22:31826-35. [cited by applicant]
Li et al., “Novel calibration method for structured-light system with an out-of-focus projector,” Appl. Opt. 53, 3415-3426 (2014). [cited by applicant]
Liu et al., “Calibration method for projector-camera based telecentric fringe projection profilometry system” Opt. Express 25, 31492-31508 (2017). [cited by applicant]
Liu et al., “Large depth-of-field 3D measurement with a microscopic structured-light system,” Opt. Commun. 481, 126540 (2021). [cited by applicant]
Meng et al., “Large DOF microscopic fringe projection profilometry with a coaxial light-field structure,” Opt. Express 30, 8015-8026 (2022). [cited by applicant]
Peng et al., “Distortion correction for microscopic fringe projection system with Scheimpflug telecentric lens” Appl. Opt. 54, 10055-10062 (2015). [cited by applicant]
Pertuz S, Puig D, Garcia MA. Analysis of focus measure operators for shape-from-focus. Pattern Recognition 2013;46:1415-32. [cited by applicant]
Proll et al., “Microscopic three-dimensional topometry with ferroelectric liquid-crystal-on-silicon displays,” Appl. Opt. 42, 1773-1778 (2003). [cited by applicant]
Ramm R, Mozaffari-Afshar M, Hohne D, Hilbert T, Speck H, Kühl S, et al. High-resolution 3D shape measurement with extended depth of field using fast chromatic focus stacking. Opt Express 2022;30:22590-607. [cited by applicant]
Rao et al., “Flexible calibration method for telecentric fringe projection profilometry systems,” Opt. Express 24, 1222-1237 (2016). [cited by applicant]
Rao L, Da F. Local blur analysis and phase error correction method for fringe projection profilometry systems. Appl Opt 2018;57:4267-76. [cited by applicant]
Su et al., “Reliability-guided phase unwrapping algorithm: a review,” Opt. Lasers Eng. 42, 245-261 (2004). [cited by applicant]
Suwajanakorn et al., “Depth from focus with your mobile phone,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3479-3506 (2015). [cited by applicant]
Szeliski R. Image alignment and stitching: a tutorial. Found Trends Comput Graph Vis 2006;2:1-104. [cited by applicant]
Van Der Jeught et al., Real-time microscopic phase-shifting profilometry, Applied Optics, vol. 54, No. 15, May 20, 2015, pp. 4953-4959. [cited by applicant]
Wang et al., “Improved performance of multi-view fringe projection 3D microscopy,” Opt. Express 25, 19408-19421 (2017). [cited by applicant]
Windecker R, Fleischer M, Tiziani HJ. Three-dimensional topometry with stereo microscopes. Opt Eng 1997;36:3372-7. [cited by applicant]
Yin et al., “Fringe projection 3D microscopy with the general imaging model,” Opt. Express 23, 6846-6857 (2015). [cited by applicant]
Zhang C, Huang PS, Chiang F-P. Microscopic phase-shifting profilometry based on digital micromirror device technology. Appl Opt 2002;41:5896-904. [cited by applicant]
Zhang et al., “Robust and efficient multi-frequency temporal phase unwrapping: optimal fringe frequency and pattern sequence selection,” Opt. Express 25, 20381-20400 (2017). [cited by applicant]
Zhang S. Absolute phase retrieval methods for digital fringe projection profilometry: A review. Opt Lasers Eng 2018;107:28-37. [cited by applicant]
Zhang S. High-Speed 3D Imaging with Digital Fringe Projection Techniques. CRC Press; 2018. [cited by applicant]
Zhong et al., “Autofocusing method for a digital fringe projection system with dual projectors,” Opt. Express 28, 12609-12620 (2020). [cited by applicant]
Zuo C, Huang L, Zhang M, Chen Q, Asundi A. Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review. Opt Lasers Eng 2016;85:84-103. [cited by applicant]