Deep neural network for hologram reconstruction with superior external generalization
A deep learning framework, termed Fourier Imager Network (FIN) is disclosed that can perform end-to-end phase recovery and image reconstruction from raw holograms of new types of samples, exhibiting success in external generalization. The FIN architecture is based on spatial Fourier transform modules with the deep neural network that process the spatial frequencies of its inputs using learnable filters and a global receptive field. FIN exhibits superior generalization to new types of samples, while also being much faster in its image inference speed, completing the hologram reconstruction task in ˜0.04 s per 1 mm 2 of the sample area. Beyond holographic microscopy and quantitative phase imaging applications, FIN and the underlying neural network architecture may open up various new opportunities to design broadly generalizable deep learning models in computational imaging and machine vision fields.
1 . A method of performing phase recovery and holographic image reconstruction from holographic amplitude images of a test sample comprising:
obtaining one or more holographic amplitude images of the test sample at one or more sample-to-sensor distances using an image sensor; and
providing a trained deep neural network that is executed using one or more processors, wherein the trained deep neural network comprises a plurality of trained spatial Fourier transform (SPAF) modules, each SPAF module configured to perform a two-dimensional (2D) Discrete Fourier transform from a spatial domain to a frequency domain followed by a linear transformation in the frequency domain followed by an inverse 2D Discrete Fourier transform to the spatial domain, wherein the trained deep neural network is trained with holographic amplitude images obtained of a plurality of training samples at one or more sample-to-sensor distances and their corresponding in-focus phase-recovered ground truth images to optimize learnable parameters in the deep neural network, wherein the trained deep neural network is configured to receive one or more holographic amplitude images of the test sample obtained with the image sensor and outputs an in-focus complex-valued output image of the test sample that includes both phase information and amplitude information of the test sample.
2 . The method of claim 1 , wherein the one or more holographic amplitude images of the test sample comprises one or more raw holographic images.
3 . The method of claim 1 , wherein the one or more holographic amplitude images of the test sample comprises one or more pixel super-resolved holographic images.
4 . The method of claim 1 , wherein the test sample and the plurality of training samples are from the same or different types of samples.
5 . The method of claim 1 , wherein each SPAF module further comprises at least one non-linear activation function.
6 . The method of claim 1 , wherein each SPAF module shares the same parameters as recursive layers.
7 . The method of claim 1 , wherein each SPAF module uses separately optimized parameters in each layer.
8 . The method of claim 1 , wherein each of the one or more holographic amplitude images of the test sample comprises a larger FOV image that is divided into smaller FOVs.
9 . The method of claim 8 , wherein the smaller FOVs are input to one or more trained deep neural networks in parallel.
10 . The method of claim 1 , wherein the test sample comprises one of tissue, cells, bacteria, viruses, fungi, pathogens, pollens, model organisms, particles, particulate matter, plastics, dust, allergens, organic or inorganic materials.
11 . A system for performing phase recovery and holographic image reconstruction from holographic amplitude images of a test sample comprising:
a microscopy device having a light source and an image sensor, the image sensor configured to capture one or more holographic amplitude images of the test sample at one or more sample-to-sensor distances; and
a computing device comprising one or more processors configured to execute software containing a trained deep neural network, wherein the trained deep neural network comprises a plurality of trained spatial Fourier transform (SPAF) modules, each SPAF module configured to perform a two-dimensional (2D) Discrete Fourier transform from a spatial domain to a frequency domain followed by a linear transformation in the frequency domain followed by an inverse 2D Discrete Fourier transform to the spatial domain, wherein the trained deep neural network is trained with holographic amplitude images obtained of a plurality of training samples at one or more sample-to-sensor distances and their corresponding in-focus phase-recovered ground truth images to optimize learnable parameters in the deep neural network, wherein the trained deep neural network is configured to receive the one or more holographic amplitude images of the test sample obtained with the image sensor and outputs an in-focus complex-valued output image of the test sample that includes phase information and amplitude information of the test sample.
12 . The system of claim 11 , wherein the test sample and the plurality of training samples are from the same or different type of samples.
13 . The system of claim 11 , further comprising a moveable stage configured to move the sample and/or the image sensor in one or more of the x, y, and z directions.
14 . The system of claim 11 , wherein each SPAF module shares the same parameters as recursive layers.
15 . The system of claim 11 , wherein each SPAF model uses separately optimized parameters in each layer.
16 . The system of claim 13 , wherein the moveable stage moves in increments in the x and y directions, and wherein the computing device further comprises image processing software configured to generate pixel super-resolved holographic images of the sample that are input to the trained deep neural network.
17 . The system of claim 11 , wherein each SPAF module further comprises at least one nonlinear activation function.
18 . The system of claim 11 , wherein each of the one or more holographic amplitude images of the test sample comprises a larger FOV image that is divided into smaller FOVs.
19 . The system of claim 18 , wherein the smaller FOVs are input to one or more trained deep neural networks in parallel.
20 . The system of claim 11 , wherein the test sample comprises one of tissue, cells, bacteria, viruses, fungi, pathogens, pollens, model organisms, particles, particulate matter, plastics, dust, allergens, organic or inorganic materials.