Large depth-of-field microscopic structured-light 3D imaging
Systems and methods of large depth-of-field microscopic structured-light 3D imaging include and/or implement capturing a plurality of images of a scene using a plurality of focus settings, each of the plurality of images corresponding to one of the plurality of focus settings, the plurality of images corresponding to a focal stack; processing the plurality of images; merging the plurality of images into an all-in-focus phase map; and reconstructing a 3D shape of the scene based on the all-in-focus phase map.
1 . A method of capturing a three-dimensional (3D) image, the method comprising:
capturing a plurality of images of a scene using a plurality of focus settings, each of the plurality of images corresponding to one of the plurality of focus settings, the plurality of images corresponding to a focal stack, wherein the operation of capturing includes capturing, by a camera, a plurality of fringe images with different frequencies;
processing the plurality of images, wherein the operation of processing includes:
computing a high-frequency phase map and a lower-frequency phase map for respective ones of the plurality of fringe images,
estimating, using the lower-frequency phase map, affine warps that minimize a phase difference between different focus settings of the camera,
applying the affine warps to the high-frequency phase map to generate an aligned fringe contrast map, and
measuring, using the aligned fringe contrast map, a focus level for a plurality of pixels of an image sensor of the camera;
merging the plurality of images into an all-in-focus phase map; and
reconstructing a 3D shape of the scene based on the all-in-focus phase map.
2 . The method of claim 1 , wherein the operation of capturing includes changing a focal length of the camera to capture multi-focus images.
3 . The method of claim 2 , wherein the camera includes an electrically tunable lens (ETL) that allows for capturing the multi-focus images without mechanical motion.
4 . The method of claim 1 , wherein the operation of merging includes stitching the high-frequency phase maps for each of the plurality of images into the all-in-focus phase map using the aligned fringe contrast map based on the measured focus levels.
5 . The method of claim 1 , further comprising determining a plurality of effective focal planes, wherein in the operation of capturing, the plurality of focus settings correspond to the plurality of effective focal planes.
6 . The method of claim 5 , further comprising capturing a focal sweep image, including changing a focal length of the camera during a camera exposure process.
7 . The method of claim 6 , further comprising generating an initial 3D point cloud based on the focal sweep image.
8 . The method of claim 7 , wherein the operation of determining includes:
calculating a maximum depth value of the scene based on the initial 3D point cloud;
calculating a minimum depth value of the scene based on the initial 3D point cloud;
generating a plurality of preliminary focal plane positions based on the maximum depth value and the minimum depth value; and
for each of the plurality of preliminary focal plane positions:
comparing an image parameter of the initial 3D point cloud to a threshold, and
in response to a determination that the image parameter exceeds the threshold, marking the preliminary focal plane position as one of the plurality of effective focal planes.
9 . The method of claim 8 , wherein each of the plurality of focus settings corresponds to one of the plurality of effective focal planes.
10 . The method of claim 1 , wherein the operation of capturing the plurality of fringe images includes capturing, by the camera, a plurality of vertical fringe images with different frequencies and a plurality of horizontal fringe images with different frequencies.
11 . An imaging system, comprising:
a projector configured to emit a projection light;
a first optical system configured to direct the projection light from the projector to a stage on which an imaging target is located;
an image sensor configured to receive an imaging light;
a second optical system configured to direct the imaging light from the stage to the image sensor; and
a controller configured to cause the system to perform operations comprising:
capturing a plurality of images of the imaging target using a plurality of focus settings, each of the plurality of images corresponding to one of the plurality of focus settings, the plurality of images corresponding to a focal stack, wherein the operation of capturing includes capturing, by the image sensor, a plurality of fringe images with different frequencies,
processing the plurality of images, wherein the operation of processing includes:
computing a high-frequency phase map and a lower-frequency phase map for respective ones of the plurality of fringe images,
estimating, using the lower-frequency phase map, affine warps that minimize a phase difference between different focus settings of the image sensor,
applying the affine warps to the high-frequency phase map to generate an aligned fringe contrast map, and
measuring, using the aligned fringe contrast map, a focus level for a plurality of pixels of the image sensor,
merging the plurality of images into an all-in-focus phase map, and
reconstructing a three-dimensional (3D) shape of the imaging target based on the all-in-focus phase map.
12 . The system of claim 11 , wherein the controller includes a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the system to perform the operations.
13 . The system of claim 11 , wherein the second optical system includes an electrically tunable lens (ETL) configured to capture multi-focus images without mechanical motion.
14 . The system of claim 11 , wherein the operation of capturing includes changing a focal length of the image sensor to capture multi-focus images.
15 . The system of claim 11 , wherein the operation of merging includes stitching the high-frequency phase maps for each of the plurality of images into the all-in-focus phase map using the aligned fringe contrast map based on the measured focus levels.
16 . The system of claim 11 , the operations further comprising capturing a focal sweep image, including changing a focal length of the image sensor during an image sensor exposure process.
17 . The system of claim 16 , the operations further comprising generating an initial 3D point cloud based on the focal sweep image.
18 . The system of claim 17 , further comprising determining a plurality of effective focal planes, wherein in the operation of capturing, the plurality of focus settings correspond to the plurality of effective focal planes, wherein the operation of determining includes:
calculating a maximum depth value of the imaging target based on the initial 3D point cloud;
calculating a minimum depth value of the imaging target based on the initial 3D point cloud;
generating a plurality of preliminary focal plane positions based on the maximum depth value and the minimum depth value; and
for each of the plurality of preliminary focal plane positions:
comparing an image parameter of the initial 3D point cloud to a threshold, and
in response to a determination that the image parameter exceeds the threshold, marking the preliminary focal plane position as one of the plurality of effective focal planes.
19 . The system of claim 18 , wherein each of the plurality of focus settings corresponds to one of the plurality of effective focal planes.
20 . The system of claim 11 , wherein the operation of capturing the plurality of fringe images includes capturing, by the image sensor, a plurality of vertical fringe images with different frequencies and a plurality of horizontal fringe images with different frequencies.