Automatic annotation of condition features in medical images
Features of a medical condition can be automatically annotated in medical images using a classification model that has been trained to classify images as having the medical condition or not. The automatically annotated features may be further processed to generate a treatment plan for treating the medical condition, for example with a laser, ultrasound, or other treatment method.
1 . A method of annotating medical images comprising:
passing a medical image to a trained machine learning (ML) classification model;
receiving from the trained ML classification model classification output comprising a confidence value that a particular condition is present in the medical image;
if the confidence in the indicated condition is above a predefined high confidence threshold, processing the medical image to automatically identify key features of the particular condition present in the medical image by:
for each of a plurality of pixel groups determining a prediction impact of changes to the respective pixel group has on the trained ML classification output; and
outputting an annotation map providing an indication of the key features of the particular condition based on the prediction impact of the plurality of pixel groups, wherein outputting the annotation map comprises:
generating a graphical user interface (GUI) comprising a representation of the annotation map; and
outputting the GUI for display on a display device,
wherein the GUI comprises functionality for automatically or semi-automatically identifying unrelated features that are not related to the particular condition and using the identified unrelated features as feedback.
2 . The method of claim 1 , wherein the predefined high confidence threshold is at least 95%.
3 . The method of claim 1 , wherein the predefined high confidence threshold is at least 99%.
4 . The method of claim 1 , wherein the GUI allows a user to select one or more key features present in the medical image and remove or modify the selected key feature from the annotation map.
5 . The method of claim 4 , wherein removing the selected key feature from the annotation map is used as feedback for adjusting the trained ML classification model.
6 . The method of claim 1 , further comprising:
processing the output annotation map to generate a treatment plan for treating the condition.
7 . The method of claim 6 , wherein processing the output annotation map comprises:
generating a treatment map based on the annotation map and including one or more treatment locations corresponding in part to one or more of the key features identified in the annotation map.
8 . The method of claim 7 , wherein generating the treatment map comprises:
retrieving one or more additional images associated with the medical image;
identifying one or more characteristics in the one or more additional images; and
determining one or more key features identified in the annotation map that are suitable for treatment based on the identified one or more characteristics in the one or more additional images.
9 . The method of claim 7 , further comprising:
generating model feedback based on a comparison of the treatment map and the annotation map; and
adjusting the trained ML classification model based on the model feedback.
10 . The method of claim 1 , further comprising:
receiving a medical image over a network from a remote computer system; and
returning the annotation map to the remote computer system.
11 . The method of claim 1 , further comprising training a classification model to provide the trained ML classification model.
12 . The method of claim 11 , wherein training the classification model comprises using data augmentation on labelled training images.
13 . The method of claim 1 , wherein determining the prediction impact of changes to the respective pixel group has on the trained ML classification output uses one or more of:
occlusion; and
saliency.
14 . The method of claim 1 , wherein at least one pixel group of the plurality of pixel groups comprises a single pixel.
15 . The method of claim 1 , wherein at least one pixel group of the plurality of pixel groups comprises a plurality of adjacent pixels.
16 . A non-transitory computer readable medium storing instructions which when executed by one or more processors of a system configure the system to provide a method annotating medical images comprising:
passing a medical image to a trained machine learning (ML) classification model;
receiving from the trained ML classification model classification output comprising a confidence value that a particular condition is present in the medical image;
if the confidence in the indicated condition is above a predefined high confidence threshold, processing the medical image to automatically identify key features of the particular condition present in the medical image by:
for each of a plurality of pixel groups determining a prediction impact of changes to the respective pixel group has on the trained ML classification output; and
outputting an annotation map providing an indication of the key features of the particular condition based on the prediction impact of the plurality of pixel groups, wherein outputting the annotation map comprises:
generating a graphical user interface (GUI) comprising a representation of the annotation map; and
outputting the GUI for display on a display device,
wherein the GUI comprises functionality for automatically or semi-automatically identifying unrelated features that are not related to the particular condition and using the identified unrelated features as feedback.
17 . A system for annotating medical images comprising:
at least one processor;
at least one memory storing instructions, which when executed by the at least one processor configure the system to provide a method of annotating medical images comprising:
passing a medical image to a trained machine learning (ML) classification model;
receiving from the trained ML classification model classification output comprising a confidence value that a particular condition is present in the medical image;
if the confidence in the indicated condition is above a predefined high confidence threshold, processing the medical image to automatically identify key features of the particular condition present in the medical image by:
for each of a plurality of pixel groups determining a prediction impact of changes to the respective pixel group has on the trained ML classification output; and
outputting an annotation map providing an indication of the key features of the particular condition based on the prediction impact of the plurality of pixel groups, wherein outputting the annotation map comprises:
generating a graphical user interface (GUI) comprising a representation of the annotation map; and
outputting the GUI for display on a display device,
wherein the GUI comprises functionality for automatically or semi-automatically identifying unrelated features that are not related to the particular condition and using the identified unrelated features as feedback.