Systems and methods for integrating tomographic image reconstruction and radiomics using neural networks

1. A method of performing computed tomography (CT) screening, diagnosis, or another CT-based image analysis tasks, comprising:

providing a training set of CT projection data and associated training diagnostic data from target individuals;

training one or more reconstruction networks to reconstruct tomographic images from the training set of CT projection data;

training one or more radiomics networks to extract features from the tomographic images and the associated training diagnostic data to diagnostically examine the target individuals;

integrating the one or more reconstruction networks and the one or more radiomics networks into an end-to-end network;

training the end-to-end network;

inputting at least a set of CT projection data from an individual to the end-to-end network; and

producing a potential diagnosis for the individual based on the features extracted by the end-to-end network.

2. The method according to claim 1 , wherein the one or more reconstruction networks include:

a first network trained to reconstruct the set of CT projection data as a rough image; and

a second network trained to refine the rough image and output the tomographic image.

3. The method according to claim 2 , wherein the first network is a network-based filtered back-projection (FBP) including at least a convolutional neural network (CNN) layer and a backprojection layer, the CNN layer including one or more filters.

4. The method according to claim 2 , wherein the second network includes at least a CNN layer including one or more filters, and a first generative adversarial network (GAN) having a discriminator which determines a loss between the reconstructed tomographic image and a true tomographic sample image.

5. The method according to claim 1 , wherein the one or more radiomics networks include:

a third network trained to identify regions of interest in the tomographic image and output a region of interest segmentation output image; and

a fourth network trained to extract features in the regions of interest and associated diagnostic data to output a feature-extracted tomographic image and identify a potential diagnosis from the set of CT projection data.

6. The method according to claim 5 , wherein the third network includes at least a CNN and a second GAN having a discriminator which determines a loss between the region of interest segmentation output image and a human annotated region of interest segmentation training image.

7. The method according to claim 5 , wherein the fourth network includes a CNN layer including one or more filters, and a third GAN having a discriminator which determines a loss between the feature-extracted tomographic image and a feature-labeled region of interest training image.

8. The method according to claim 7 , wherein the fourth network is trained in an unsupervised fashion with region of interest segmentation output images to remove noise and/or artifacts, and learn to classify based on extracted features, via transfer learning, or combinations thereof.

9. The method according to claim 8 , wherein the fourth network is fine-tuned in a supervised fashion after the unsupervised training, transfer learning, or combinations thereof.

10. The method according to claim 1 , wherein the associated training diagnostic data includes pathological results, histological findings, clinical reports, laboratory tests, genetic profiling, genomic assays, proteomic assays, or combinations thereof.

11. A system for performing screening, diagnosis, or another image-based analysis tasks, comprising:

a non-transitory computer storage medium coupled with a computer device including one or more processors, the one or more processors configured to execute one or more computer programs encoded on the non-transitory computer storage medium for examining an individual, the one or more computer programs including:

a trained end-to-end network, including:

a reconstruction module including one or more reconstruction algorithms and/or networks trained to reconstruct tomographic images from a set of tomographic data from the individual; and

a radiomics module integrated with the reconstruction module, the radiomics module including one or more radiomics networks trained to extract features in the tomographic images to diagnostically examine the individual.

12. The system according to claim 11 , further comprising a network training module including tomographic training data, associated training diagnostic data, true tomographic sample images, human annotated region of interest segmentation training images, and feature-labeled region of interest training images.

13. The system according to claim 12 , wherein the one or more reconstruction networks include:

a first network trained to reconstruct the tomographic data as a rough image, the first network including a network-based FBP including at least a CNN layer and a backprojection layer; and

a second network trained to refine the rough image and output the tomographic image, the second network including at least a CNN layer or a sparsely-connected layer.

14. The system according to claim 12 , wherein the one or more radiomics networks include:

a third network trained to identify regions of interest in the tomographic image and output a region of interest segmentation output image, the third network including at least a CNN layer or a sparsely connected layer; and

a fourth network trained to extract features in the regions of interest and associated diagnostic data to output a feature-extracted tomographic image and identify a potential diagnosis for a set of tomographic data, the fourth network including at least a CNN layer or a sparsely-connected layer.

15. The system according to claim 13 , wherein the second network includes a GAN trained with at least some true tomographic sample images.

16. The system according to claim 14 , wherein the third network includes a GAN trained with at least some human annotated region of interest segmentation training images.

17. The system according to claim 14 , wherein the fourth network includes a GAN trained firstly in an unsupervised fashion with region of interest segmentation output images from the third network to remove noise and/or artifacts, and learn to classify based on extracted features, and subsequently in a supervised fashion with the feature-labeled region of interest training images, diagnostic training data, or combinations thereof, to extract features from the regions of interest.

18. The method according to claim 12 , wherein the tomographic training data includes CT projection data, MRI data, nuclear imaging data, ultrasound signals, optical data, or combinations thereof.

19. A method of performing screening, diagnosis, or another image-based analysis tasks, comprising:

providing a set of tomographic training data and associated training diagnostic data from target individuals;

training one or more networks to extract features from the tomographic training data and associated training diagnostic data from target individuals;

integrating the one or more networks into a unified end-to-end network;

training the unified network with the tomographic training data and the associated training diagnostic data from the target individuals;

inputting a set of tomographic data from an individual to the unified end-to-end network; and

producing a potential diagnosis for the individual based on the features extracted by the unified end-to-end network.

20. The method according to claim 19 , wherein the tomographic training data includes CT projection data, Mill data, nuclear imaging data, ultrasound signals, optical data, or combinations thereof.