ML-based methods for pseudo-CT and HR MR image estimation
The present disclosure describes a computer-implemented method of transforming a low-resolution MR image to a high-resolution MR image using a deep CNN-based MRI SR network and a computer-implemented method of transforming an MR image to a pseudo-CT (sCT) image using a deep CNN-based sCT network. The present disclosure further describes a MR image-guided radiation treatment system that includes a computing device to implement the MRI SR and CT networks and to produce a radiation plan based in the resulting high resolution MR images and sCT images.
1. A computer-implemented method of transforming a low-resolution MR image into a super-resolution MR image using an MRI SR deep CNN system comprising a deep CNN-based de-noising auto-encoder (DAE) network and a deep CNN-based super-resolution generative network (SRG), the method comprising:
receiving, using a computing device, a low-resolution MR image;
transforming, using the computing device, the low-resolution MR image into a de-noised MR image using the DAE network, the DAE network comprising six convolutional encoder layers with 4×4 filters and six de-convolutional decoder layers with 4×4 filters, wherein each convolutional encoder layer comprises a single convolutional filter with stride 2, each de-convolution decoder layer comprises a single deconvolutional filter with stride 2, and each convolutional encoder layer and each de-convolution decoder layer ends with a leaky and standard rectified linear unit (ReLU); and,
transforming, using the computing device, the de-noised MR image into the super-resolution MR image using the SRG network.
2. The computer-implemented method of claim 1 , wherein the SRG network comprises:
two up-sampling layers,
eight residual blocks, each residual block comprising two 3×3 convolutional filters separated by a ReLU activation with an elementwise sum operator attached at the end of the layer; and
two output layers, each output layer comprising a 3×3 convolutional filter, ReLU activation, and a subpixel operator up-sampling layer.
3. The computer-implemented method of claim 2 , further comprising training the DAE network by:
receiving, using the computing device, a set of noisy low resolution MR images;
transforming, using the computing device, each noisy MR image into a de-noised MR image using a noise filter, wherein each noisy MR image and corresponding de-noised MR image together form a noisy/de-noised MR image pair; and
training, using the computing device, the DAE network to minimize a reconstruction error given by ∥g θ g (f θ f ({tilde over (x)}))−x∥ for each matched noisy/de-noised low resolution image pair, where x denotes each de-noised MR image, {tilde over (x)} denotes each noisy MR image, and f θ f and g θ g denote the encoding and decoding network parameterized by θ f and θ g , respectively.
4. The computer-implemented method of claim 3 , wherein the noise filter comprises a non-local means filter.
5. The computer-implemented method of claim 2 , further comprising training the SRG network by:
receiving, using the computing device, a set of matched low resolution/high resolution MR image pairs;
forming, using the computing device, a generative adversarial network (GAN) including a generative model G parametrized by θ G and comprising the SRG network and a discriminative model D parametrized by θ D , the discriminative model D configured to determine a probability that a high resolution MR image is a high resolution image or an SRG-transformed low resolution MR image from a matched low resolution/high resolution MR image pair; and
training, using the computing device, the GAN to solve the optimization problem given by
min
θ
G
max
θ
D
{
𝔼
x
~
P
data
log
D
θ
D
(
x
)
+
𝔼
z
~
p
z
log
(
1
-
D
θ
D
(
G
θ
G
(
z
)
)
)
}
by updating D and G in alternating steps while fixing the other parameter, wherein the GAN is trained if D is unable to determine whether each high resolution MR image is the selected high resolution MR image or the transformed low resolution MR image from each matched low resolution/high resolution MR image pair.
6. The computer-implemented method of claim 5 , wherein the set of matched low resolution/high resolution MR image pairs is produced by:
transforming, using the computing device, a high resolution MR image to a low resolution MR image training using a deep CNN-based down-sampling network (DSN), the DSN comprising:
two down-sampling layers, each down-sampling layer comprising a 3×3 convolutional filter of stride 2 followed by a ReLU activation;
two residual blocks, each residual block comprising two 3×3 convolutional filters separated by a ReLU activation and followed by an elementwise sum operator; and
an output layer.