Deep learning based image reconstruction
Disclosed herein are systems, methods, and instrumentalities associated with reconstructing magnetic resonance (MR) images based on under-sampled MR data. The MR data include 2D or 3D information, and may encompass multiple contrasts and multiple coils. The MR images are reconstructed using deep learning (DL) methods, which may accelerate the scan and/or image generation process. Challenges imposed by the large quantity of the MR data and hardware limitations are overcome by separately reconstructing MR images based on respective subsets of contrasts, coils, and/or readout segments, and then combining the reconstructed MR images to obtain desired multi-contrast results.
1. A method of reconstructing magnetic resonance (MR) images, the method comprising:
obtaining an under-sampled MR dataset associated with an anatomical structure, wherein the under-sampled MR dataset is associated with a single MR scan and includes data associated with multiple contrast settings, multiple coils, and multiple segments in a readout direction;
determining a first portion of the under-sampled MR dataset that corresponds to a first target contrast, wherein the first portion of the under-sampled MR dataset includes data associated with a first subset of the multiple coils and a first segment in the readout direction;
reconstructing, using one or more neural networks, a first MR image of the anatomical structure based on the first portion of the under-sampled MR dataset;
determining a second portion of the under-sampled MR dataset that corresponds to the first target contrast, wherein the second portion of the under-sampled MR dataset includes data associated with a second subset of the multiple coils and a second segment in the readout direction;
reconstructing, using the one or more neural networks, a second MR image of the anatomical structure based on the second portion of the under-sampled MR dataset; and
deriving an MR image of the anatomical structure that is associated with the first target contrast based at least on the first MR image and the second MR image.
2. The method of claim 1 , wherein the first MR image is reconstructed independently from the second MR image.
3. The method of claim 1 , wherein the first MR image and the second MR image are reconstructed sequentially, and wherein the second MR image is reconstructed based on the first MR image.
4. The method of claim 1 , further comprising:
determining a third portion of the under-sampled MR dataset that corresponds to a second target contrast, wherein the third portion of the under-sampled MR dataset includes data associated with a third subset of the multiple coils and a third segment in the readout direction;
reconstructing, using the one or more neural networks, a third MR image of the anatomical structure based on the third portion of the under-sampled MR dataset;
determining a fourth portion of the under-sampled MR dataset that corresponds to the second target contrast, wherein the fourth portion of the under-sampled MR dataset includes data associated with a fourth subset of the multiple coils and a fourth segment in the readout direction;
reconstructing, using the one or more neural networks, a fourth MR image of the anatomical structure based on the fourth portion of the under-sampled MR dataset; and
deriving an MR image of the anatomical structure that is associated with the second target contrast based at least on the third MR image and the fourth MR image.
5. The method of claim 4 , wherein the MR image associated with the first target contrast and the MR image of the anatomical structure associated with the second target contrast are derived independently of each other.
6. The method of claim 1 , wherein the one or more neural networks comprise a three-dimensional convolutional neural network (CNN).
7. The method of claim 1 , wherein the one or more neural networks comprise a cascade of convolutional neural network (CNN) blocks, and wherein each CNN block is associated with a respective data consistency layer.
8. The method of claim 1 , wherein the one or more neural networks comprise a plurality of three-dimensional depthwise separable convolutional layers.
9. The method of claim 1 , wherein the under-sampled MR dataset comprises two-dimensional or three-dimensional MR data.
10. An apparatus, comprising:
one or more processors configured to:
obtain an under-sampled MR dataset associated with an anatomical structure, wherein the under-sampled MR dataset is a single MR scan and includes data associated with multiple contrast settings, multiple coils, and multiple segments in a readout direction;
determine a first portion of the under-sampled MR dataset that corresponds to a first target contrast, wherein the first portion of the under-sampled MR dataset includes data associated with a first subset of the multiple coils and a first segment in the readout direction;
reconstruct, using one or more neural networks, a first MR image of the anatomical structure based on the first portion of the under-sampled MR dataset;
determine a second portion of the under-sampled MR dataset that corresponds to the first target contrast, wherein the second portion of the under-sampled MR dataset includes data associated with a second subset of the multiple coils and a second segment in the readout direction;
reconstruct, using the one or more neural networks, a second MR image of the anatomical structure based on the second portion of the under-sampled MR data; and
derive an MR image of the anatomical structure that is associated with the first target contrast based at least on the first MR image and the second MR image.
11. The apparatus of claim 10 , wherein the first MR image is reconstructed independently from the second MR image.
12. The apparatus of claim 10 , wherein the first MR image and the second MR image are reconstructed sequentially, and wherein the second MR image is reconstructed based on the first MR image.
13. The apparatus of claim 10 , wherein the one or more processors are further configured to:
determine a third portion of the under-sampled MR dataset that corresponds to a second target contrast, wherein the third portion of the under-sampled MR dataset includes data associated with a third subset of the multiple coils and a third segment in the readout direction;
reconstruct, using the one or more neural networks, a third MR image of the anatomical structure based on the third portion of the under-sampled MR dataset;
determine a fourth portion of the under-sampled MR dataset that corresponds to the second target contrast, wherein the fourth portion of the under-sampled MR dataset includes data associated with a fourth subset of the multiple coils and a fourth segment in the readout direction;
reconstruct, using the one or more neural networks, a fourth MR image of the anatomical structure based on the fourth portion of the under-sampled MR dataset; and
derive an MR image of the anatomical structure that is associated with the second target contrast based at least on the third MR image and the fourth MR image.
14. The apparatus of claim 13 , wherein the MR image associated with the first target contrast and the MR image of the anatomical structure associated with the second target contrast are derived independently of each other.
15. The apparatus of claim 10 , wherein the one or more neural networks comprise a three-dimensional convolutional neural network (CNN).
16. The apparatus of claim 10 , wherein the one or more neural networks comprise a cascade of convolutional neural network (CNN) blocks, and wherein each CNN block is associated with a respective data consistency layer.
17. The apparatus of claim 10 , wherein the one or more neural networks comprise a plurality of three-dimensional depthwise separable convolutional layers.
18. The apparatus of claim 10 , wherein the under-sampled MR dataset comprises two-dimensional or three-dimensional MR data.
19. The apparatus of claim 10 , wherein the one or more neural networks each have a structure determined via a neural architecture search.
20. A method of training a neural network to learn a model for reconstructing magnetic resonance (MR) images, the method comprising:
the neural network obtaining a first portion of a MR training dataset, wherein the MR training dataset is associated with a single MR scan and comprises under-sampled MR data associated with an anatomical structure, multiple contrast settings, multiple coils, and multiple segments in a readout direction, and wherein the first portion of the MR training dataset that corresponds to a target contrast and includes data associated with a first subset of the multiple coils and a first segment in the readout direction;
the neural network reconstructing a first MR image of the anatomical structure based on the first portion of the MR training dataset;
the neural network obtaining a second portion of the MR training dataset, wherein the second portion of the MR training dataset is associated with the target contrast and includes data associated with a second subset of the multiple coils and a second segment in the readout direction;
the neural network reconstructing a second MR image of the anatomical structure based on the second portion of the MR training dataset;
the neural network deriving an MR image of the anatomical structure that is associated with the target contrast based at least on the first MR image and the second MR image; and
the neural network adjusting one or more operating parameters based on a difference between the derived MR image and a ground truth MR image.