System and method for 3D imaging reconstruction using dual-domain neural network
This application describes a method and system for medical 3D imaging reconstruction based on limited-angle 2D image acquisition, which only requires 30˜50% of the data required by traditional scanner-based 3D reconstruction. This approach significantly reduces the radiation dose requirement and improves clinical efficiency. An example method includes: capturing a plurality of 2-dimensional (2D) projections of a target that cover a limited angle of the target; calibrating the plurality of 2D projections of the target to obtain a plurality of calibrated 2D projections; generating a sinogram based on the plurality of calibrated 2D projections; inputting the sinogram into a dual-domain neural network to obtain a 3D volumetric image of the target that is geometrically corrected and calibrated.
1 . A computer-implemented method, comprising:
capturing a plurality of 2-dimensional (2D) projections of a target that cover a limited angle of the target;
performing geometric calibration and physical correction on the plurality of 2D projections of the target to obtain a plurality of physically corrected and geometrically calibrated 2D projections; and
generating a sinogram with a limited angle coverage of the target based on the plurality of physically corrected and geometrically calibrated 2D projections;
updating the sinogram by at least inserting estimated data to a missing angle region of the sinogram;
inputting the updated sinogram into a dual-domain neural network, wherein the dual-domain neural network comprises:
a first neural network trained to optimize a sinogram constructed based on limited angle fluoroscopic images, and
a second neural network trained to optimize a 3D reconstruction that is generated based on the sinogram optimized by the first neural network; and
obtaining a 3D volumetric image of the target as an output of the dual-domain neural network.
2 . The computer-implemented method of claim 1 , wherein the updating the sinogram by at least inserting estimated data to the missing angle region of the sinogram comprises:
generating a first 3-dimensional (3D) reconstruction of the target based on the sinogram with limited angle coverage;
performing a physics-based forward projection on the first 3D reconstruction to obtain the estimated data to the missing angle region; and
inserting the estimated data to the missing angle region of the sinogram to obtain the updated sinogram.
3 . The computer-implemented method of claim 1 , wherein the inputting the updated sinogram into the dual-domain neural network comprises:
feeding the updated sinogram into the first neural network trained to optimize the data inserted to the missing angle region of the sinogram, thereby obtaining an optimized sinogram;
generating a second 3D reconstruction based on the optimized sinogram;
feeding the second 3D reconstruction into the second neural network trained to further optimizing the second 3D reconstruction by suppressing artifacts in the second 3D reconstruction; and
obtaining the 3D volumetric image of the target as an output of the second neural network.
4 . The computer-implemented method of claim 3 , wherein the suppressing artifacts comprises adding, removing, or shuffling pixels of the second 3D reconstruction.
5 . The computer-implemented method of claim 3 , wherein the generating the second 3D reconstruction based on the optimized sinogram comprises:
if prior information of the target is available, performing iterative reconstruction to generate the second 3D reconstruction by using both the prior information of the target and the optimized sinogram,
if the prior information of the target is not available, performing Filtered Back Projection (FBP) based on the optimized sinogram to generate the second 3D reconstruction,
wherein the prior information of the target comprises a shape or a pattern of a physical object used during the capturing of the plurality of 2D projections, a prior computerized tomography (CT) scan of the target, or real-time optical tracking data from one or more optical sensors tracking the device that captures the plurality of 2D projections.
6 . The computer-implemented method of claim 1 , wherein the geometric calibration performed on the plurality of 2D projections of the target comprises:
spatially shifting the plurality of 2D projections based on prior information of the target to achieve a circular trajectory among the plurality of 2D projections, wherein the prior information comprises a shape or a pattern of a physical object used during the capturing of the plurality of 2D projections, a prior computerized tomography (CT) scan of the target, or real-time optical tracking data from one or more optical sensors tracking the device that captures the plurality of 2D projections.
7 . The computer-implemented method of claim 1 , wherein the method further comprises jointly training the first neural network and the second neural network, wherein the jointly training comprises:
obtaining training samples, each training sample comprising (1) a ground truth sinogram of a training subject constructed based on a full-angle scanning, (2) a ground truth 3D reconstruction of the training subject constructed based on the ground truth sinogram of the training subject; and (3) a preliminary sinogram constructed based on the ground truth sinogram;
inputting the preliminary sinogram to the first neural network to generate an optimized sinogram;
constructing a training 3D reconstruction based on the optimized sinogram;
inputting the training 3D reconstruction to the second neural network to generate an optimized 3D reconstruction;
backpropagating a first loss between the optimized 3D reconstruction and the ground truth 3D reconstruction for updating parameters of the second neural network;
backpropagating a second loss between the optimized sinogram and the ground truth sinogram for updating parameters of the first neural network; and
iteratively performing the jointly training until both the first loss and the second loss are below respective thresholds.
8 . The computer-implemented method of claim 7 , wherein the obtaining the training samples comprises:
constructing the preliminary sinogram by removing a portion of data from the ground truth sinogram.
9 . The computer-implemented method of claim 8 , wherein the removing the portion of data comprises:
removing between 50% to 90% of data from the ground truth sinogram.
10 . The computer-implemented method of claim 7 , wherein the jointly training further comprises a plurality of self-augmented training stages, each self-augmented training stage comprising:
generating a new training sample during the jointly training using the preliminary sinogram and the optimized 3D reconstruction generated by the second neural network; and
feeding the new training sample into the dual-domain neural network for a new round of training.
11 . The computer-implemented method of claim 10 , wherein the generating the new training sample comprises:
performing forward projection on the optimized 3D reconstruction and filling in missing data to the preliminary sinogram based on output of the forward projection.
12 . The computer-implemented method of claim 1 , further comprises:
aligning the 3D volumetric image of the target with images of a navigation system by using a plurality of optical reference points, thereby obtaining aligned 3D images of the target; and
displaying the aligned 3D images of the target on a screen in real-time to provide 3D navigation for surgical procedures involving probing tools, or for verifying interventional radiology procedures.
13 . The computer-implemented method of claim 1 , wherein the method further comprises:
obtaining geometrical parameters of an imaging system capturing the plurality of 2D projections of the target by using one or more optical tracking sensors, and
wherein the geometric calibration performed on the plurality of 2D projections of the target comprises:
spatially shifting the plurality of 2D projections based on the geometrical parameters of the imaging system to achieve a circular trajectory among the plurality of 2D projections.
14 . A non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising:
capturing a plurality of 2-dimensional (2D) projections of a target that cover a limited angle of the target;
performing geometric calibration and physical correction on the plurality of 2D projections of the target to obtain a plurality of physically corrected and geometrically calibrated 2D projections; and
generating a sinogram with a limited angle coverage of the target based on the plurality of physically corrected and geometrically calibrated 2D projections;
updating the sinogram by at least inserting estimated data to a missing angle region of the sinogram;
inputting the updated sinogram into a dual-domain neural network, wherein the dual-domain neural network comprises:
a first neural network trained to optimize a sinogram constructed based on limited angle fluoroscopic images, and
a second neural network trained to optimize a 3D reconstruction that is generated based on the sinogram optimized by the first neural network; and
obtaining a 3D volumetric image of the target as an output of the dual-domain neural network.
15 . The non-transitory computer-readable storage medium of claim 14 , wherein the updating the sinogram by at least inserting estimated data to the missing angle region of the sinogram comprises:
generating a first 3-dimensional (3D) reconstruction of the target based on the sinogram with limited angle coverage;
performing a physics-based forward projection on the first 3D reconstruction to obtain the estimated data to the missing angle region; and
inserting the estimated data to the missing angle region of the sinogram to obtain the updated sinogram.
16 . The non-transitory computer-readable storage medium of claim 14 , wherein the inputting the updated sinogram into the dual-domain neural network comprises:
feeding the updated sinogram into the first neural network trained to optimize the data inserted to the missing angle region of the sinogram, thereby obtaining an optimized sinogram;
generating a second 3D reconstruction based on the optimized sinogram;
feeding the second 3D reconstruction into the second neural network trained to further optimizing the second 3D reconstruction by suppressing artifacts in the second 3D reconstruction; and
obtaining the 3D volumetric image of the target as an output of the second neural network.
17 . The non-transitory computer-readable storage medium of claim 14 , wherein the operations further comprise jointly training the first neural network and the second neural network, the jointly training comprising:
obtaining training samples, each training sample comprising (1) a ground truth sinogram of a training subject constructed based on a full-angle scanning, (2) a ground truth 3D reconstruction of the training subject constructed based on the ground truth sinogram of the training subject; and (3) a preliminary sinogram constructed based on the ground truth sinogram;
inputting the preliminary sinogram to the first neural network to generate an optimized sinogram;
constructing a training 3D reconstruction based on the optimized sinogram;
inputting the training 3D reconstruction to the second neural network to generate an optimized 3D reconstruction;
backpropagating a first loss between the optimized 3D reconstruction and the ground truth 3D reconstruction for updating parameters of the second neural network;
backpropagating a second loss between the optimized sinogram and the ground truth sinogram for updating parameters of the first neural network; and
iteratively performing the jointly training until both the first loss and the second loss are below respective thresholds.
18 . The non-transitory computer-readable storage medium of claim 14 , wherein the physical correction comprises performing real-time scatter estimation using graphic processing unit (GPU)-based real-time Monte Carlo Simulation.
19 . The non-transitory computer-readable storage medium of claim 14 , wherein the operations further comprise:
aligning the 3D volumetric image of the target with images of a navigation system by using a plurality of optical reference points, thereby obtaining aligned 3D images of the target; and
displaying the aligned 3D images of the target through a graphic user interface (GUI) to provide real-time guidance for navigating a lead wire trajectory during a procedure, confirming a placement of a medical tool, or diagnostic assistance.
20 . A system, comprising one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions executable by the one or more processors to cause the system to perform operations comprising:
capturing a plurality of 2-dimensional (2D) projections of a target that cover a limited angle of the target;
performing geometric calibration and physical correction on the plurality of 2D projections of the target to obtain a plurality of physically corrected and geometrically calibrated 2D projections; and
generating a sinogram with a limited angle coverage of the target based on the plurality of physically corrected and geometrically calibrated 2D projections;
updating the sinogram by at least inserting estimated data to a missing angle region of the sinogram;
inputting the updated sinogram into a dual-domain neural network, wherein the dual-domain neural network comprises:
a first neural network trained to optimize a sinogram constructed based on limited angle fluoroscopic images, and
a second neural network trained to optimize a 3D reconstruction that is generated based on the sinogram optimized by the first neural network; and
obtaining a 3D volumetric image of the target as an output of the dual-domain neural network.