Fluorescence lifetime imaging using deep learning
One embodiment provides an apparatus for fluorescence lifetime imaging (FLI). The apparatus includes a deep neural network (DNN). The DNN includes a first convolutional layer, a plurality of intermediate layers and an output layer. The first convolutional layer is configured to receive FLI input data. Each intermediate layer is configured to receive a respective intermediate input corresponding to an output of a respective prior layer. Each intermediate layer is further configured to provide a respective intermediate output related to the received respective intermediate input. The output layer is configured to provide estimated FLI output data corresponding to the received FLI input data. The DNN is trained using synthetic data.
1. An apparatus for fluorescence lifetime imaging (FLI), the apparatus comprising:
a deep neural network (DNN) comprising:
a first convolutional layer configured to receive FLI input data;
a plurality of intermediate layers, each intermediate layer configured to receive a respective intermediate input corresponding to an output of a respective prior layer, each intermediate layer further configured to provide a respective intermediate output related to the received respective intermediate input; and
an output layer configured to provide estimated FLI output data corresponding to the received FLI input data, the output layer is a fully convolutional (FC) down-sample layer comprising:
a first two-dimensional (2D) convolutional block;
a first intermediate block coupled to an output of the first 2D convolutional block;
a second 2D convolutional block coupled to an output of the first intermediate block;
a second intermediate block coupled to an output of the second 2D convolutional block;
a third 2D convolutional block coupled to an output of the second intermediate block; and
a rectified linear unit block coupled to an output of the third 2D convolutional block;
wherein the first intermediate block and the second intermediate block each comprise a batch normalization function and a rectified linear unit;
wherein the DNN is trained using synthetic data.
2. The apparatus of claim 1 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a 3D residual block, a reshape layer, a two-dimensional (2D) convolutional layer, and at least one 2D residual block.
3. The apparatus of claim 1 , wherein the first convolutional layer is a separable two-dimensional (2D) convolutional layer; the plurality of intermediate layers comprises a 2D exception block, a 2D convolutional layer, and a 2D residual block.
4. The apparatus of claim 1 , further comprising a discriminator network configured to compare estimated training output data with training synthetic output data during training, the DNN and the discriminator network corresponding to a generative adversarial network (GAN) during training.
5. The apparatus of claim 1 , wherein the DNN further comprises a second convolutional layer configured to receive optical property data, the plurality of intermediate layers comprises a concatenate layer, and the estimated FLI output data is further related to the optical property data.
6. The apparatus of claim 1 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a second 3D convolutional layer, a reshape layer, a separable two-dimensional (2D) convolutional layer, and a 2D exception block, and the DNN comprises a plurality of output layers and each output layer is a coefficient block that corresponds to the fully convolutional (FC) down-sample layer.
7. The apparatus of claim 1 , wherein the FLI input data is selected from the group comprising visible FLI microscopy (FLIM) data, near infrared (NIR) FLIM data and NIR gated macroscopy FLI (MFLI) data.
8. A method for fluorescence lifetime imaging (FLI), the method comprising:
receiving, by a first convolutional layer of a deep neural network (DNN), FLI input data;
receiving, by each intermediate layer of a plurality of intermediate layers, a respective intermediate input corresponding to an output of a respective prior layer;
providing, by each intermediate layer, a respective intermediate output related to the received respective intermediate input; and
providing, by an output layer, estimated FLI output data corresponding to the received FLI input data, the output layer is a fully convolutional (FC) down-sample layer comprising:
a first two-dimensional (2D) convolutional block;
a first intermediate block coupled to an output of the first 2D convolutional block;
a second 2D convolutional block coupled to an output of the first intermediate block;
a second intermediate block coupled to an output of the second 2D convolutional block;
a third 2D convolutional block coupled to an output of the second intermediate block; and
a rectified linear unit block coupled to an output of the third 2D convolutional block;
wherein the first intermediate block and the second intermediate block each comprise a batch normalization function and a rectified linear unit wherein the DNN is trained using synthetic data.
9. The method of claim 8 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a 3D residual block, a reshape layer, a two-dimensional (2D) convolutional layer, and at least one 2D residual block.
10. The method of claim 8 , wherein the first convolutional layer is a separable two-dimensional (2D) convolutional layer; the plurality of intermediate layers comprises a 2D exception block, a 2D convolutional layer.
11. The method of claim 8 , further comprising comparing, by a discriminator network, estimated training output data with training synthetic output data during training, the DNN and the discriminator network corresponding to a generative adversarial network (GAN) during training.
12. The method of claim 8 , further comprising receiving, by a second convolutional layer, optical property data, the plurality of intermediate layers comprising a concatenate layer, and the estimated FLI output data is further related to the optical property data.
13. The method of claim 8 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a second 3D convolutional layer, a reshape layer, a separable two-dimensional (2D) convolutional layer, and a 2D exception block, and the DNN comprises a plurality of output layers and each output layer is a coefficient block that corresponds to the fully convolutional (FC) down-sample layer.
14. The method of claim 8 , wherein the FLI input data is selected from the group comprising visible FLI microscopy (FLIM) data, near infrared (NIR) FLIM data and NIR gated macroscopy FLI (MFLI) data.
15. A fluorescence lifetime imaging (FLI) deep learning system, the system comprising:
a processor;
a memory;
input/output circuitry; and
a deep neural network (DNN), the DNN comprising:
a first convolutional layer configured to receive FLI input data;
a plurality of intermediate layers, each intermediate layer configured to receive a respective intermediate input corresponding to an output of a respective prior layer, each intermediate layer further configured to provide a respective intermediate output related to the received respective intermediate input; and
an output layer configured to provide estimated FLI output data corresponding to the received FLI input data, the output layer is a fully convolutional (FC) down-sample layer comprising:
a first two-dimensional (2D) convolutional block;
a first intermediate block coupled to an output of the first 2D convolutional block;
a second 2D convolutional block coupled to an output of the first intermediate block;
a second intermediate block coupled to an output of the second 2D convolutional block;
a third 2D convolutional block coupled to an output of the second intermediate block; and
a rectified linear unit block coupled to an output of the third 2D convolutional block;
wherein the first intermediate block and the second intermediate block each comprise a batch normalization function and a rectified linear unit;
wherein the DNN is trained using synthetic data.
16. The system of claim 15 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a 3D residual block, a reshape layer, a two-dimensional (2D) convolutional layer, and at least one 2D residual block.
17. The system of claim 15 , wherein the first convolutional layer is a separable two-dimensional (2D) convolutional layer; the plurality of intermediate layers comprises a 2D exception block, a 2D convolutional layer, and a 2D residual block.
18. The system of claim 15 , further comprising a discriminator network configured to compare estimated training output data with training synthetic output data during training, the DNN and the discriminator network corresponding to a generative adversarial network (GAN) during training.
19. The system of claim 15 , wherein the DNN further comprises a second convolutional layer configured to receive optical property data, the plurality of intermediate layers comprises a concatenate layer, and the estimated FLI output data is further related to the optical property data.
20. The system of claim 15 , wherein the first convolutional layer is a three-dimensional (3D) convolutional layer, the plurality of intermediate layers comprises a second 3D convolutional layer, a reshape layer, a separable two-dimensional (2D) convolutional layer, and a 2D exception block, and the DNN comprises a plurality of output layers and each output layer is a coefficient block that corresponds to the fully convolutional (FC) down-sample layer.