UNSUPERVISED CONTRASTIVE LEARNING FOR DEFORMABLE AND DIFFEOMORPHIC MULTIMODALITY IMAGE REGISTRATION
A computer-implemented method that includes providing as input to the neural network, a first image and a second image. The method further includes obtaining, using the neural network, a transformed image based on the first image that may be aligned with the second image. The method further includes obtaining a plurality of first patches from the transformed image by encoding the transformed image using a first encoder that has a first plurality of encoding layers. The method further includes obtaining a plurality of second patches from the second image by encoding the second image using a second encoder that has a second plurality of encoding layers. The method further includes computing a loss value based on comparison of respective first patches and second patches. The method further includes adjusting one or more parameters of the neural network based on the loss value.
1 . A computer-implemented method to train a neural network to perform image registration, the method comprising:
providing as input to the neural network, a first image and a second image;
obtaining, using the neural network, a transformed image based on the first image that is aligned with the second image;
obtaining a plurality of first patches from the transformed image by encoding the transformed image using a first encoder that has a first plurality of encoding layers, wherein one or more patches of the first plurality of patches are obtained from different layers of the first plurality of encoding layers;
obtaining a plurality of second patches from the second image by encoding the second image using a second encoder that has a second plurality of encoding layers, wherein at least two patches of a second plurality of patches are obtained from different layers of the second plurality of encoding layers;
computing a loss value based on comparison of respective first patches and second patches; and
adjusting one or more parameters of the neural network based on the loss value.
2 . The computer-implemented method of claim 1 , wherein before training the neural network to perform image registration, further comprising:
training the first encoder and the second encoder with joint L1+Local Normalized Cross Correlation (LNCC) loss functions; and
freezing parameters of the first encoder and the second encoder.
3 . The computer-implemented method of claim 1 , further comprising:
training the neural network using a hyperparameter for each loss function by randomly sampling from a uniform distribution during training.
4 . The computer-implemented method of claim 3 , wherein an increase in the hyperparameter results in the neural network outputting a smoother displacement field and a decrease in the hyperparameter results in a deformed first image that is more closely aligned to the second image.
5 . The computer-implemented method of claim 1 , wherein the neural network outputs a displacement field and further comprising:
applying, with a spatial transform network, the displacement field to the first image, wherein the spatial transform network outputs the transformed image.
6 . The computer-implemented method of claim 1 , wherein computing the loss value based on comparison of respective first patches and second patches includes:
extracting, with the first encoder and the second encoder, multi-scale features for the respective first patches and second patches; and
applying a loss function based on a comparison of the multi-scale features to determine the loss value.
7 . The computer-implemented method of claim 6 , wherein multilayer perceptrons are applied to the multi-scale features.
8 . The computer-implemented method of claim 6 , wherein the loss function maximizes the multi-scale features and uses a global mutual information loss on image intensity histograms.
9 . The computer-implemented method of claim 1 , wherein training the neural network is an unsupervised process.
10 . The computer-implemented method of claim 1 , wherein different layers of the first plurality of encoding layers correspond to different scales of the transformed image.
11 . A device to perform image registration, the device comprising:
one or more processors; and
a memory coupled to the one or more processors, with instructions stored thereon that, when executed by the processor, cause the one or more processors to perform operations comprising:
providing a first image of a first type and a second image of a second type, different from the first type, as input to a trained neural network;
obtaining, as output of the trained neural network, a displacement field for the first image; and
obtaining a transformed image by applying the displacement field to the first image via a spatial transform network, wherein corresponding features of the transformed image and the second image are aligned.
12 . The device of claim 11 , wherein the trained neural network employs a hyperparameter.
13 . The device of claim 12 , wherein an increase in the hyperparameter results in the trained neural network outputting a smoother displacement field.
14 . The device of claim 12 , wherein a decrease in the hyperparameter results in a deformed first image that is more closely aligned to the second image.
15 . The device of claim 11 , wherein the first image and the second image are of a human tissue or a human organ.
16 . The device of claim 11 , wherein the transformed image is output for viewing on a display.
17 . A non-transitory computer-readable medium to train a neural network to perform image registration with instructions stored thereon that, when executed by a processor, cause the processor to perform operations, the operations comprising:
providing as input to the neural network, a first image and a second image;
obtaining, using the neural network, a transformed image based on the first image that is aligned with the second image;
obtaining a plurality of first patches from the transformed image by encoding the transformed image using a first encoder that has a first plurality of encoding layers, wherein one or more patches of the first plurality of patches are obtained from different layers of the first plurality of encoding layers;
obtaining a plurality of second patches from the second image by encoding the second image using a second encoder that has a second plurality of encoding layers, wherein at least two patches of a second plurality of patches are obtained from different layers of the second plurality of encoding layers;
computing a loss value based on comparison of respective first patches and second patches; and
adjusting one or more parameters of the neural network based on the loss value.
18 . The computer-readable medium of claim 17 , wherein before training the neural network to perform image registration, the operations further comprise:
training the first encoder and the second encoder with joint L1+Local Normalized Cross Correlation (LNCC) loss functions; and
freezing parameters of the first encoder and the second encoder.
19 . The computer-readable medium of claim 17 , wherein the operations further comprise:
training the neural network using a hyperparameter for each loss function by randomly sampling from a uniform distribution during training.
20 . The computer-readable medium of claim 19 , wherein an increase in the hyperparameter results in the neural network outputting a smoother displacement field and a decrease in the hyperparameter results in a deformed first image that is more closely aligned to the second image.