Real-time ultra-quality multi-parametric four-dimensional magnetic resonance imaging system and the method thereof
A computer-implemented method for training a convolutional neural network (CNN) using training data comprising a pair of original and downsampled 4D magnetic resonance imaging (MRI) data is provided. The CNN is used to generate multi-parametric 4D magnetic resonance (MR) images based on multi-parametric 3D MR images in real-time. The method includes receiving a 4D MR image formed by a plurality of fixed images of different frames; converting the plurality of fixed images into a plurality of k-space data by non-uniform fast Fourier transform (NUFFT); applying radial scan to the k-space data to simulate real-time MR image acquisition, and generating a plurality of downsampled fixed images by inverse NUFFT; training a CNN with training data comprising the 4D MR image and the corresponding downsampled 4D MR image; and estimating the multi-parametric 4D MR image in real-time by applying apply the predicted DVF to the multi-parametric 3D MR images.
1. A computer-implemented method for training a convolutional neural network (CNN) using training data comprising a pair of original and downsampled 4D magnetic resonance imaging (MRI) data, the CNN used to generate multi-parametric 4D magnetic resonance (MR) images based on multi-parametric 3D MR images in real-time, the multi-parametric 4D MR images having an improved tumor contrast-to-noise ratio (CNR), an improved organ edges sharpness, and a reduced perceptual blur metric (PBM), the method comprising:
receiving, by a processor, a 4D MR image formed by a plurality of fixed images of different frames;
converting, by the processor, the plurality of fixed images into a plurality of k-space data by non-uniform fast Fourier transform (NUFFT), wherein the plurality of k-space data is an array of numbers representing spatial frequencies of the plurality of fixed images in a Cartesian coordinate system;
applying, by the processor, radial scan to the plurality of k-space data to simulate real-time MR image acquisition; and generating a plurality of downsampled fixed images by inverse NUFFT (iNUFFT) for forming a corresponding downsampled 4D MR image;
reconstructing, by the processor, the plurality of downsampled fixed images using a densely connected network in a reconstruction branch of a dual-supervised downsampling-invariant deformable registration (D3R) model to obtain reconstructed fixed images with enhanced image details and suppressed artifacts;
training, by a machine learning method, the CNN with training data comprising the 4D MR image and the corresponding downsampled 4D MR image and the reconstructed fixed images, wherein the CNN is the D3R model comprising the reconstruction branch and a registration branch and is trained to calculate a predicted displacement vector field (DVF) based on deformable registration of the reconstructed fixed images and the 4D MR image by the registration branch; and
estimating, by the processor, the multi-parametric 4D MR image in real-time by applying the predicted DVF to the multi-parametric 3D MR images.
2. The method of claim 1 further comprising dually supervising the step of training the CNN by an end-to-end point error (EPE) and a normalized correlation coefficient (NCC).
3. The method of claim 2 further comprising generating a reference DVF from the 4D MR image using conventional registration methods as a ground truth DVF, wherein the conventional registration methods comprise a parametric total variation (pTV) algorithm.
4. The method of claim 3 , wherein the step of dually supervising the step of training the CNN further comprises determining a first loss function between the reference DVF and the predicted DVF as the EPE; and determining a second loss function between a warped image and the 4D MR image as the NCC.
5. The method of claim 1 , wherein the step of training the CNN comprises:
deformably registering, by the registration branch, the reconstructed fixed images and the 4D MR image to predict deformations between the reconstructed fixed images and the 4D MR image for determining the predicted DVF.
6. The method of claim 5 , wherein:
the densely connected network comprises a plurality of layers each having a dense block and a transition layer; and
an individual layer receives inputs from outputs of all previous layers of the plurality of layers.
7. The method of claim 6 , wherein the step of reconstructing the plurality of downsampled fixed images further comprises performing convolution and pooling, wherein the densely connected network is characterized with a feed-forward characteristic for enhancing image details and suppressing artifacts.
8. The method of claim 5 , wherein the step of deformably registering the reconstructed fixed image and the 4D MR image further comprises pairing and mapping the reconstructed fixed image and the 4D MR image to a DVF pair for alignment, wherein the registration branch is a supervised registration framework based on the CNN.
9. The method of claim 1 , wherein the multi-parametric 3D MR images comprise T1-weighted (T1w) MRI, T2-weighted (T2w) MRI, and diffusion-weighted MR imaging (DWI).
10. A four-dimensional (4D) magnetic resonance imaging (MRI) system for generating multi-parametric 4D magnetic resonance (MR) images based on multi-parametric 3D MR images in real-time, the multi-parametric 4D MR images having an improved tumor contrast-to-noise ratio (CNR), an improved organ edges sharpness, and a reduced perceptual blur metric (PBM), the system comprising:
one or more computer devices collectively programmed with a data pre-processing module, a model training module, and an image generating module, wherein:
the data pre-processing module is configured to downsample a 4D MR image by non-uniform fast Fourier transform (NUFFT) and generate a corresponding downsampled 4D MR image, wherein:
the 4D MR image is formed by a plurality of fixed images of different frames,
the plurality of fixed images are converted into a plurality of k-space data by the non-uniform fast Fourier transform (NUFFT),
the plurality of k-space data is an array of numbers representing spatial frequencies of the plurality of fixed images in a Cartesian coordinate system,
radial scan is applied to the plurality of k-space data to simulate real-time MR image acquisition, and
a plurality of downsampled fixed images are generated by inverse NUFFT (iNUFFT) for forming the corresponding downsampled 4D MR image;
the data pre-processing module is further configured to reconstruct the plurality of downsampled fixed images using a densely connected network in a reconstruction branch of a dual-supervised downsampling-invariant deformable registration (D3R) model to obtain reconstructed fixed images with enhanced image details and suppressed artifacts;
the model training module is configured to train a convolutional neural network (CNN) with training data using a machine learning method, wherein the CNN is the D3R model comprising the reconstruction branch and a registration branch, the training data comprises the 4D MR image, the corresponding downsampled 4D MR image and the reconstructed fixed images from the data pre-processing module, and the CNN is trained to calculate a predicted displacement vector field (DVF) based on deformable registration of the reconstructed fixed images and the 4D MR image by the registration branch; and
the image generation module is configured to apply the predicted DVF to the multi-parametric 3D MR images to estimate the multi-parametric 4D MR image in real-time,
wherein:
the training data for the CNN comprises the 4D MR image and the corresponding downsampled 4D MR image from the data pre-processing module;
the model training module is dually supervised by an end-to-end point error (EPE) and a normalized correlation coefficient (NCC); and
the model training module is configured to perform deformation registration of the corresponding downsampled 4D MR image to train the CNN.
11. The system of claim 10 , wherein the 4D MR image is a moving image formed by the plurality of fixed images.
12. The system of claim 10 , wherein the reconstruction branch is configured to reconstruct the plurality of downsampled fixed images using a densely connected network to obtain reconstructed fixed images, wherein:
the densely connected network comprises a plurality of layers each having a dense block and a transition layer; and
an individual layer receives inputs from outputs of all previous layers of the plurality of layers.
13. The system of claim 12 , wherein the transition layer is configured to perform convolution and pooling; and the densely connected network is characterized with a feed-forward characteristic for enhancing image details and suppressing artifacts.
14. The system of claim 12 , wherein the registration branch receives the reconstructed fixed images and the 4D MR image for predicting deformations between the reconstructed fixed images and the 4D MR image, thereby the registration branch generates the predicted DVF for the image generation module.
15. The system of claim 14 , wherein the registration branch is a supervised registration framework based on the CNN, wherein the reconstructed fixed images and the 4D MR image are paired and mapped to a DVF pair for alignment.
16. The system of claim 10 , wherein the EPE determines a first loss function between a reference DVF and the predicted DVF; and the NCC determines a second loss function between a warped image and the 4D MR image.
17. The system of claim 16 , wherein the reference DVF is a ground truth DVF computed from the 4D MR image using conventional registration methods for supervising the model training module, wherein the conventional registration methods comprise a parametric total variation (pTV) algorithm.