Localization and classification of abnormalities in medical images
Systems and methods are provided for classifying an abnormality in a medical image. An input medical image depicting a lesion is received. The lesion is localized in the input medical image using a trained localization network to generate a localization map. The lesion is classified based on the input medical image and the localization map using a trained classification network. The classification of the lesion is output. The trained localization network and the trained classification network are jointly trained.
1. A method for classifying a lesion in a medical image, comprising:
receiving a plurality of input medical images acquired with different acquisition protocols;
localizing a lesion in each of the plurality of input medical images using a trained localization network to generate a localization map for each of the plurality of input medical images;
combining the localization maps for the plurality of input medical images;
classifying the lesion based on the plurality of input medical images and the combined localization maps using a trained classification network; and
outputting the classification of the lesion,
wherein the trained localization network and the trained classification network are jointly trained.
2. The method of claim 1 , further comprising:
jointly training the trained localization network and the trained classification network by separately training the localization network to determine weights of the localization network during a first training phase, and training the classification network based on the weights of the localization network during a second training phase.
3. The method of claim 2 , wherein separately training the localization network comprises:
receiving a multi-site dataset associated with different clinical sites and a deployment dataset associated with a deployment clinical site;
training a deep learning model based on the multi-site dataset; and
optimizing the trained deep learning model based on the deployment dataset to provide the trained localization network.
4. The method of claim 1 , wherein:
the plurality of input medical images is of a multi-parametric magnetic resonance imaging (mpMRI) image.
5. The method of claim 4 , further comprising:
preprocessing the plurality of input medical images of the mpMRI image to address variances between the plurality of input medical images.
6. The method of claim 5 , wherein preprocessing the plurality of input medical images of the mpMRI image to address variances between the plurality of input medical images comprises:
removing geometric variability in the plurality of input medical images of the mpMRI image.
7. The method of claim 5 , wherein preprocessing the plurality of input medical images of the mpMRI image to address variances between the plurality of input medical images comprises:
normalizing intensity variability in the plurality of input medical images of the mpMRI image.
8. The method of claim 1 , wherein the localization maps are associated with a score of the lesion.
9. The method of claim 1 , further comprising:
generating a clinical relevance map and a label indicating whether the lesion is clinically significant based on the localization maps and one or more of: a size of the lesion, an average intensity within the lesion, a variance of intensities within the lesion, radiomic features, and lexicon based features.
10. The method of claim 1 , further comprising:
generating a patch from the plurality of input medical images using the localization maps; and
generating a score associated with the lesion using a trained machine learning network.
11. An apparatus for classifying a lesion in a medical image, comprising:
means for receiving a plurality of input medical images acquired with different acquisition protocols;
means for localizing a lesion in each of the plurality of input medical images using a trained localization network to generate a localization map for each of the plurality of input medical images;
means for combining the localization maps for the plurality of input medical images;
means for classifying the lesion based on the plurality of input medical images and the combined localization maps using a trained classification network; and
means for outputting the classification of the lesion,
wherein the trained localization network and the trained classification network are jointly trained.
12. The apparatus of claim 11 , further comprising:
means for jointly training the trained localization network and the trained classification network by separately training the localization network to determine weights of the localization network during a first training phase, and training the classification network based on the weights of the localization network during a second training phase.
13. The apparatus of claim 12 , wherein the means for separately training the localization network comprises:
means for receiving a multi-site dataset associated with different clinical sites and a deployment dataset associated with a deployment clinical site;
means for training a deep learning model based on the multi-site dataset; and
means for optimizing the trained deep learning model based on the deployment dataset to provide the trained localization network.
14. The apparatus of claim 11 , further comprising:
means for generating a clinical relevance map and a label indicating whether the lesion is clinically significant based on the localization maps and one or more of:
a size of the lesion, an average intensity within the lesion, a variance of intensities within the lesion, radiomic features, and lexicon based features.
15. The apparatus of claim 11 , further comprising:
means for generating a patch from the plurality of input medical images using the localization maps; and
means for generating a score associated with the lesion using a trained machine learning network.
16. A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:
receiving a plurality of input medical images acquired with different acquisition protocols;
localizing a lesion in each of the plurality of input medical images using a trained localization network to generate a localization map for each of the plurality of input medical images;
combining the localization maps for the plurality of input medical images;
classifying the lesion based on the plurality of input medical images and the combined localization maps using a trained classification network; and
outputting the classification of the lesion,
wherein the trained localization network and the trained classification network are jointly trained.
17. The non-transitory computer readable medium of claim 16 , wherein:
the plurality of input medical images is of a multi-parametric magnetic resonance imaging (mpMRI) image.
18. The non-transitory computer readable medium of claim 17 , the operations further comprising:
preprocessing the plurality of input medical images of the mpMRI image to address variances between the plurality of input medical images.
19. The non-transitory computer readable medium of claim 18 , wherein preprocessing the plurality of input medical images of the mpMRI image to address variances between the plurality of input medical images comprises:
removing geometric variability in the plurality of input medical images of the mpMRI image.
20. The non-transitory computer readable medium of claim 18 , wherein preprocessing the plurality of input medical images of the mpMRI image to address variances between the plurality of input medical images comprises:
normalizing intensity variability in the plurality of input medical images of the mpMRI image.