Neural network for generating synthetic medical images
Systems, computer-implemented methods, and computer readable media for generating a synthetic image of an anatomical portion based on an origin image of the anatomical portion acquired by an imaging device using a first imaging modality are disclosed. These systems may be configured to receive the origin image of the anatomical portion acquired by the imaging device using the first imaging modality, receive a convolutional neural network model trained for predicting the synthetic image based on the origin image, and convert the origin image to the synthetic image through the convolutional neural network model. The synthetic image may resemble an imaging of the anatomical portion using a second imaging modality differing from the first imaging modality.
1 . A computer-implemented method for training a convolutional neural network for producing a synthetic image, the method comprising:
receiving a plurality of training origin images acquired using a first imaging modality, wherein the plurality of training origin images include respective images from a first plane and from a second plane, wherein the first plane and the second plane are different;
receiving a plurality of training destination images acquired using a second imaging modality, wherein the first imaging modality and the second imaging modality are different types of modalities, wherein the plurality of training destination images include respective images from the first plane and from the second plane, and wherein each training destination image corresponds to a respective training origin image that provides a corresponding view of one or more anatomical objects;
training a set of model parameters of the convolutional neural network using the plurality of training origin images and corresponding images of the plurality of training destination images; and
outputting the convolutional neural network in a trained model, the trained model configured to receive one or more patient images acquired using the first imaging modality and output one or more synthetic patient images that resemble imaging from the second imaging modality.
2 . The method of claim 1 , wherein the first imaging modality and the second imaging modality are different types of modalities provided from among: Magnetic Resonance Imaging, Computed Tomography, Ultrasound Imaging, Positron Emission Tomography, or Single-Photon Emission Computed Tomography.
3 . The method of claim 1 , wherein the plurality of training origin images include multi-channel images acquired using different acquisition channels, and wherein the multi-channel images include T1-weighted Magnetic Resonance images and T2-weighted Magnetic Resonance images.
4 . The method of claim 1 , wherein the plurality of training origin images are provided in a stack of two-dimensional images or a three-dimensional volume, and wherein the plurality of training destination images are provided in a corresponding stack of two-dimensional images or a corresponding three-dimensional volume.
5 . The method of claim 1 , wherein the convolutional neural network comprises a first and a second convolutional neural network, wherein the first convolutional neural network is trained with two-dimensional images of the plurality of training origin images from the first plane, and wherein the second convolutional neural network is trained with two-dimensional images of the plurality of training origin images from the second plane.
6 . The method of claim 1 , wherein the convolutional neural network includes:
a first layer configured to determine, for a respective training origin image of the plurality of training origin images, a feature map; and
a second layer configured to determine a respective training destination image from the feature map.
7 . The method of claim 6 , wherein the first layer includes a plurality of encoding layers and the second layer includes a plurality of decoding layers.
8 . The method of claim 6 , wherein the first layer is configured to reduce a size of the feature map through down-sampling, and the second layer is configured to increase a size of the feature map through up-sampling.
9 . The method of claim 1 , wherein the convolutional neural network includes a plurality of convolutional layers, and wherein the set of model parameters include learnable filter weights used by the plurality of convolutional layers.
10 . The method of claim 1 , wherein training the set of model parameters of the convolutional neural network further includes:
converting the plurality of training origin images to a plurality of synthetic images using the convolutional neural network;
determining a difference between the plurality of synthetic images and the corresponding images of the plurality of training destination images, wherein the difference is measured by a loss function calculated based on the plurality of synthetic images and the corresponding images of the plurality of training destination images; and
updating the set of model parameters based on the difference.
11 . A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to train a convolutional neural network for producing a synthetic image, wherein the instructions cause the at least one processor to:
identify a plurality of training origin images acquired using a first imaging modality, wherein the plurality of training origin images include respective images from a first plane and from a second plane, wherein the first plane and the second plane are different;
identify a plurality of training destination images acquired using a second imaging modality, wherein the first imaging modality and the second imaging modality are different types of modalities, wherein the plurality of training destination images include respective images from the first plane and from the second plane, and wherein each training destination image corresponds to a respective training origin image that provides a corresponding view of one or more anatomical objects;
train a set of model parameters of the convolutional neural network using the plurality of training origin images and corresponding images of the plurality of training destination images; and
output the convolutional neural network in a trained model, the trained model configured to receive one or more patient images acquired using the first imaging modality and output one or more synthetic patient images that resemble imaging from the second imaging modality.
12 . The computer-readable medium of claim 11 , wherein the first imaging modality and the second imaging modality are different types of modalities provided from among: Magnetic Resonance Imaging, Computed Tomography, Ultrasound Imaging, Positron Emission Tomography, or Single-Photon Emission Computed Tomography.
13 . The computer-readable medium of claim 11 , wherein the plurality of training origin images include multi-channel images acquired using different acquisition channels, and wherein the multi-channel images include T1-weighted Magnetic Resonance images and T2-weighted Magnetic Resonance images.
14 . The computer-readable medium of claim 13 , wherein the plurality of training origin images are provided in a stack of two-dimensional images or a three-dimensional volume, and wherein the plurality of training destination images are provided in a corresponding stack of two-dimensional images or a corresponding three-dimensional volume.
15 . The computer-readable medium of claim 11 , wherein the convolutional neural network comprises a first and a second convolutional neural network, wherein the first convolutional neural network is trained with two-dimensional images of the plurality of training origin images from the first plane, and wherein the second convolutional neural network is trained with two-dimensional images of the plurality of training origin images from the second plane.
16 . The computer-readable medium of claim 11 , wherein the convolutional neural network includes:
a first layer configured to determine, for a respective training origin image of the plurality of training origin images, a feature map; and
a second layer configured to determine a respective training destination image from the feature map.
17 . The computer-readable medium of claim 16 , wherein the first layer includes a plurality of encoding layers and the second layer includes a plurality of decoding layers.
18 . The computer-readable medium of claim 17 , wherein the first layer is configured to reduce a size of the feature map through down-sampling, and the second layer is configured to increase a size of the feature map through up-sampling.
19 . The computer-readable medium of claim 11 , wherein the convolutional neural network includes a plurality of convolutional layers, and wherein the set of model parameters include learnable filter weights used by the plurality of convolutional layers.
20 . The computer-readable medium of claim 11 , wherein to train the set of model parameters of the convolutional neural network includes operations to:
convert the plurality of training origin images to a plurality of synthetic images using the convolutional neural network;
determine a difference between the plurality of synthetic images and the corresponding images of the plurality of training destination images, wherein the difference is measured by a loss function calculated based on the plurality of synthetic images and the corresponding images of the plurality of training destination images; and
update the set of model parameters based on the difference.