Identifying abnormal images in set of medical images based on combined feature set with global features extracted using first CNN and local features extracted using second CNN analyzing high resolution image patches
Disclosed herein is a method and system for identifying abnormal images in a set of medical images for optimal assessment of the medical images. A plurality of global features from each medical image is extracted based on pretrained weights associated with each global feature. Similarly, plurality of local features from each medical image is extracted analyzing a predefined number of image patches generated from a higher resolution image corresponding to each medical image. Further, an abnormality score for each medical image is determined based on weights associated with a combined feature set obtained by concatenating the plurality of global features and the plurality of local features. Thereafter, the medical image is identified as an abnormal image when the abnormality score of the medical image is higher than a predefined first threshold score.
1. A computer-implemented method of identifying abnormal images in a set of medical images for optimal assessment of the medical images, the method comprising:
extracting a plurality of global features from each medical image of the set of medical images based on pretrained weights associated with each of the plurality of global features, wherein the global features are assigned the pretrained weights using a first Convolutional Neural Network (CNN);
extracting, using a second Convolutional Neural Network (CNN) different from the first CNN, a plurality of local features from each medical image by analysing a predefined number of image patches corresponding to each medical image, wherein the predefined number of image patches are generated by obtaining a higher resolution image corresponding to each medical image and splitting each higher resolution image into the predefined number of image patches, wherein the respective image patches are analyzed to extract local features from the respective image patches, and wherein the extracted local features for the respective image patches are combined to obtain the plurality of local features, wherein a resolution of the medical image is less than a resolution of the higher resolution image;
determining an abnormality score for each medical image based on weights associated with a combined feature set obtained by concatenating the plurality of global features and the plurality of local features, wherein the abnormality score is determined using a pretrained feature classifier; and
identifying the medical image as an abnormal image when the abnormality score of the medical image is higher than a predefined first threshold score.
2. The method of claim 1 , wherein the plurality of global features and the plurality of local features are extracted using a pretrained feature extraction model.
3. The method of claim 2 , wherein training the feature extraction model comprises:
obtaining a plurality of train medical images;
extracting ground truth labels corresponding to each of the plurality of train medical images, wherein the ground truth labels comprises one or more system-assigned ground truth labels and corresponding one or more expert-assigned labels;
comparing the one or more system-assigned ground truth labels with the corresponding one or more expert-assigned labels to determine the image classes having consensus;
extracting one or more system-predicted ground truth labels from image classes having no consensus;
comparing the one or more system-predicted ground truth labels with the corresponding system-assigned ground truth labels to determine train medical images having matching ground truth labels; and
providing the one or more matching ground truth labels and the corresponding train medical images for training the feature extraction model.
4. The method of claim 1 , wherein determining the abnormality score for each medical image comprises comparing the weights associated with the combined feature set with corresponding stored weights associated with the pretrained feature classifier.
5. The method of claim 1 , further comprising:
classifying the abnormal image as a critical abnormal image when the abnormality score of the image is more than a predefined second threshold score, wherein the predefined second threshold score is greater than the predefined first threshold score; and
classifying the abnormal image as a challenging abnormal image when the abnormality score of the abnormal image is more than the predefined first threshold score and less than the predefined second threshold score.
6. The method of claim 1 , wherein identifying the abnormal image further comprises:
dynamically creating a worklist comprising each of the abnormal images; and
assigning the worklist to an expert analyst for assessment of the abnormal images including one or more critical abnormal images and one or more challenging abnormal images.
7. The method of claim 6 , wherein each of the one or more challenging abnormal images are provided to an expert analyst for assessment and each of the one or more critical abnormal images are flagged as priority and provided to the expert analyst for immediate assessment.
8. An automated assessment system for identifying abnormal images in a set of medical images for optimal assessment of the medical images, the automated system comprising:
a processor; and
a memory, communicatively coupled to the processor, wherein the memory stores processor-executable instructions, which on execution, cause the processor to:
extract a plurality of global features from each medical image of the set of medical images based on pretrained weights associated with each of the plurality of global features;
extract, using a second Convolutional Neural Network (CNN) different from the first CNN, a plurality of local features from each medical image by analysing a predefined number of image patches corresponding to each medical image, wherein the predefined number of image patches are generated by obtaining a higher resolution image corresponding to each medical image and splitting each higher resolution image into the predefined number of image patches, wherein the respective image patches are analyzed to extract local features from the respective image patches, and wherein the extracted local features for the respective image patches are combined to obtain the plurality of local features, wherein a resolution of the medical image is less than a resolution of the higher resolution image;
determine an abnormality score for each medical image based on weights associated with a combined feature set obtained by concatenating the plurality of global features and the plurality of local features, wherein the abnormality score is determined using a pretrained feature classifier; and
identify the medical image as an abnormal image when the abnormality score of the medical image is higher than a predefined first threshold score.
9. The automated assessment system of claim 8 , wherein the plurality of global features and the plurality of local features are extracted using a pretrained feature extraction model.
10. The automated assessment system of claim 9 , wherein the processor trains the feature extraction model by:
obtaining a plurality of train medical images;
extracting ground truth labels corresponding to each of the plurality of train medical images, wherein the ground truth labels comprises one or more system-assigned ground truth labels and corresponding one or more expert-assigned labels;
comparing the one or more system-assigned ground truth labels with the corresponding one or more expert-assigned labels to determine the image classes having consensus;
extracting one or more system-predicted ground truth labels from image classes not having consensus;
comparing the one or more system-predicted ground truth labels with the corresponding system-assigned ground truth labels to determine one or more images having matching ground truth labels; and
providing the one or more matching ground truth labels and corresponding the train medical images for training the feature extraction model.
11. The automated assessment system of claim 8 , wherein the processor determines the abnormality score for each medical image by comparing the weights associated with the combined feature set with corresponding stored weights associated with the pretrained feature classifier.
12. The automated assessment system of claim 8 , wherein the processor is further configured to:
classify the abnormal image as a critical abnormal image when the abnormality score of the image is more than a predefined second threshold score, wherein the predefined second threshold score is greater than the predefined first threshold score; and
classify the abnormal image as a challenging abnormal image when the abnormality score of the abnormal image is more than the predefined first threshold score and less than the predefined second threshold score.
13. The automated assessment system of claim 8 , wherein identifying the abnormal image further comprises:
dynamically creating a worklist comprising each abnormal image; and
assigning the worklist to an expert analyst for assessment of the abnormal images including one or more critical abnormal images and one or more challenging abnormal images.
14. The automated assessment system of claim 13 , wherein each of the one or more challenging abnormal images are provided to an expert analyst for assessment and each of the one or more critical abnormal images are flagged as priority and provided to the expert analyst for immediate assessment.
15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to identify abnormal images in a set of medical images for optimal assessment of the medical images, comprising:
extracting a plurality of global features from each medical image of the set of medical images based on pretrained weights associated with each of the plurality of global features, wherein the global features are assigned the pretrained weights using a first Convolutional Neural Network (CNN);
extracting, using a second Convolutional Neural Network (CNN) different from the first CNN, a plurality of local features from each medical image by analysing a predefined number of image patches corresponding to each medical image, wherein the predefined number of image patches are generated by obtaining a higher resolution image corresponding to each medical image and splitting each higher resolution image into the predefined number of image patches, wherein the respective image patches are analyzed to extract local features from the respective image patches, and wherein the extracted local features for the respective image patches are combined to obtain the plurality of local features, wherein a resolution of the medical image is less than a resolution of the higher resolution image;
determining an abnormality score for each medical image based on weights associated with a combined feature set obtained by concatenating the plurality of global features and the plurality of local features, wherein the abnormality score is determined using a pretrained feature classifier; and
identifying the medical image as an abnormal image when the abnormality score of the medical image is higher than a predefined first threshold score.