Systems and methods for cross-modality image segmentation

1. A system for segmenting a medical image, comprising:

a communication interface configured to receive the medical image acquired by an image acquisition device;

a memory configured to store a plurality of learning networks jointly trained using first training images of a first imaging modality and second training images of a second imaging modality,

wherein the plurality of learning networks are jointly trained with a cross-modality learning network using a loss function constructed using outputs by the cross-modality learning network and the plurality of learning networks; and

a processor, configured to segment the medical image using a segmentation network selected from the plurality of learning networks.

2. The system of claim 1 , wherein the learning network is a convolutional neural network.

3. The system of claim 1 , wherein the first imaging modality and the second imaging modality are selected from a group of Computed Tomography (CT), magnetic resonance imaging (MRI), functional MRI, cone beam computed tomography (CBCT), positron emission tomography (PET), single-photon emission computed tomography (SPECT), X-ray, optical tomography, fluorescence imaging, ultrasound imaging, and radiotherapy portal imaging.

4. The system of claim 1 , wherein the communication interface is further configured to receive the plurality of learning networks from a training device, wherein the plurality of learning networks include a first-modality learning network and a second-modality learning network, wherein the training device is configured to jointly train the cross-modality learning network, the first-modality learning network, and the second-modality learning network,

wherein the cross-modality learning network includes a convolutional base network, and an auxiliary network for learning image features from the first training images and second training images,

wherein the first-modality learning network includes the convolutional base network, and a first classification network for segmenting the first training images, and

wherein the second-modality learning network includes the convolutional base network, and a second classification network for segmenting the second training images.

5. The system of claim 4 , wherein the processor is further configured to:

determine that the medical image is of the first imaging modality; and

select the first-modality learning network as the segmentation network for segmenting the medical image.

6. The system of claim 4 , wherein the training device is further configured to:

register the first training images with the second training images;

select first image patches from first training images and second image patches from the second training images;

pair the first image patches with the second image patches based on the registration; and

jointly train the cross-modality learning network, the first-modality learning network, and the second-modality learning network using the paired image patches.

7. The system of claim 6 , wherein the training device is further configured to determine a label for each pair of first image patch and second image patch, the label indicating a distance between physical positions of the first image patch and the second image patch of the pair.

8. The system of claim 6 , wherein the training device is further configured to apply the image features learned by an auxiliary network to register the first training images with the second training images.

9. The system of claim 4 , wherein to jointly train the networks, the training device is configured to use the loss function, wherein the loss function is constructed using outputs by the cross-modality learning network, the first-modality learning network, and the second-modality learning network.

10. A method for segmenting a medical image, comprising:

receiving the medical image acquired by an image acquisition device;

receiving, from a training device, a plurality of learning networks jointly trained using first training images of a first imaging modality and second training images of a second imaging modality,

wherein the plurality of learning networks are jointly trained with a cross-modality learning network using a loss function constructed using outputs by the cross-modality learning network and the plurality of learning networks; and

segmenting, by a processor, the medical image using a segmentation network selected from the plurality of learning networks.

11. The method of claim 10 , wherein the learning network is a convolutional neural network.

12. The method of claim 10 , wherein the first imaging modality and the second imaging modality are selected from a group of Computed Tomography (CT), magnetic resonance imaging (MRI), functional MRI, cone beam computed tomography (CBCT), positron emission tomography (PET), single-photon emission computed tomography (SPECT), X-ray, optical tomography, fluorescence imaging, ultrasound imaging, and radiotherapy portal imaging.

13. The method of claim 10 , wherein the plurality of learning networks include a first-modality learning network and a second-modality learning network, wherein the method further comprises jointly training the cross-modality learning network, the first-modality learning network, and the second-modality learning network,

wherein the cross-modality learning network includes a convolutional base network, and an auxiliary network for learning image features from the first training images and second training images,

wherein the first-modality learning network includes the convolutional base network, and a first classification network for segmenting the first training images, and

wherein the second-modality learning network includes the convolutional base network, and a second classification network for segmenting the second training images.

14. The method of claim 13 , further comprising:

determining that the medical image is of the first imaging modality; and

selecting the first-modality learning network as the segmentation network for segmenting the medical image.

15. The method of claim 13 , wherein jointly training the networks further comprises:

registering the first training images with the second training images;

selecting first image patches from first training images and second image patches from the second training images;

pairing the first image patches with the second image patches based on the registration; and

jointly training the cross-modality learning network, the first-modality learning network, and the second-modality learning network using the paired image patches.

16. The method of claim 15 , wherein jointly training the networks further comprises determining a label for each pair of first image patch and second image patch, the label indicating a distance between physical positions of the first image patch and the second image patch of the pair.

17. The system of claim 15 , wherein registering the first training images with the second training images further comprises applying the image features learned by an auxiliary network to improve the registration.

18. The method of claim 13 , wherein jointly training the networks further comprises using the loss function, wherein the loss function is constructed using outputs by the cross-modality learning network, the first-modality learning network, and the second-modality learning network.

19. A non-transitory computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by at least one processor, performs a method for segmenting a medical image, the method comprising:

receiving the medical image acquired by an image acquisition device;

receiving a plurality of learning networks jointly trained using first training images of a first imaging modality and second training images of a second imaging modality,

wherein the plurality of learning networks are jointly trained with a cross-modality learning network using a loss function constructed using outputs by the cross-modality learning network and the plurality of learning networks; and

segmenting the medical image using a segmentation network selected from the plurality of learning networks.

20. The non-transitory computer-readable medium of claim 19 , wherein the plurality of learning networks include a first-modality learning network and a second-modality learning network, wherein the method further comprises jointly training the cross-modality learning network, the first-modality learning network, and the second-modality learning network,

wherein the cross-modality learning network includes a convolutional base network, and an auxiliary network for learning image features from the first training images and second training images,

wherein the first-modality learning network includes the convolutional base network, and a first classification network for segmenting the first training images, and

wherein the second-modality learning network includes the convolutional base network, and a first classification network for segmenting the second training images.