Co-training framework to mutually improve concept extraction from clinical notes and medical image classification
A system and method for training a text report identification machine learning model and an image identification machine learning model, including: initially training a text report machine learning model, using a labeled set of text reports including text pre-processing the text report and extracting features from the pre-processed text report, wherein the extracted features are input into the text report machine learning model; initially training an image machine learning model, using a labeled set of images; applying the initially trained text report machine learning model to a first set of unlabeled text reports with associated images to label the associated images; selecting a first portion of labeled associated images; re-training the image machine learning model using the selected first portion of labeled associated images; applying the initially trained image machine learning model to a first set of unlabeled images with associated text reports to label the associated text reports; selecting a first portion of labeled associated text reports; and re-training the text report machine learning model using the selected first portion of labeled associated text reports.
1. A system for training a text report identification machine learning model and an image identification machine learning model, comprising:
a memory;
a processor connected to the memory, the processor configured to:
initially train a text report machine learning model, using a labeled set of text reports including text pre-processing the text report and extracting features from the pre-processed text report, wherein the extracted features are input into the text report machine learning model;
initially train an image machine learning model, using a labeled set of images;
apply the initially trained text report machine learning model to a first set of unlabeled text reports with associated images to label the associated images;
select a first portion of labeled associated images;
re-train the image machine learning model using the selected first portion of labeled associated images;
apply the initially trained image machine learning model to a first set of unlabeled images with associated text reports to label the associated text reports;
select a first portion of labeled associated text reports; and
re-train the text report machine learning model using the selected first portion of labeled associated text reports.
2. The system of 1 , wherein
selecting a portion of labeled associated images includes selecting associated images with text report machine learning model outputs having a confidence level above a first confidence threshold, and
selecting a portion of labeled associated text reports includes selecting associated text reports with image machine learning model outputs having a confidence level above a second confidence threshold.
3. The system of claim 1 , wherein
selecting a portion of labeled associated images further includes selecting the N associated images with text report machine learning model outputs having the highest confidence levels, wherein N is a predetermined value, and
selecting a portion of labeled associated text reports further includes selecting the M associated text reports with image machine learning model outputs having the highest confidence levels, wherein M is a predetermined value.
4. The system of claim 1 , wherein
selecting a portion of labeled associated images includes selecting associated images with text report machine learning model outputs having a confidence level above a first confidence threshold,
when there are more than N selected associated images, further selecting N associated images with text reports that have the highest confidence levels,
selecting a portion of labeled associated text reports includes selecting associated text reports with image machine learning model outputs having a confidence level above a second confidence threshold, and
when there are more than M selected associated text reports, further selecting M associated text reports with images that have the highest confidence levels.
5. The system of claim 1 , wherein the outputs of the text report machine learning model and image machine learning model indicated the same set of classification concepts.
6. The system of claim 1 , wherein
the image machine learning model is re-trained until the initially trained text report machine learning model is applied to all the unlabeled text reports with associated images, and
the text report machine learning model is re-trained until the initially trained image machine learning model is applied to all the unlabeled images with associated text reports.
7. The system of claim 1 , wherein
the image machine learning model is re-trained until the image machine learning model performance on validation set of input image data does not improve, or
the text report machine learning model is re-trained until the text report machine learning model performance on a validation set of input text report data does not improve.
8. The system of 1 , wherein the processor is further configured to:
apply the retrained text report machine learning model to a second set of unlabeled text reports with associated images to label the associated images;
select a second portion of labeled associated images;
re-train the retrained image machine learning model using the selected second portion of labeled associated images;
apply the retrained image machine learning model to a second set of unlabeled images with associated text reports to label the associated text reports;
select a second portion of labeled associated text reports; and
re-train the retrained text report machine learning model using the selected second portion of labeled associated text reports.
9. A method for training a text report identification machine learning model and an image identification machine learning model, comprising:
initially training a text report machine learning model, using a labeled set of text reports including text pre-processing the text report and extracting features from the pre-processed text report, wherein the extracted features are Input into the text report machine learning model;
initially training an Image machine learning model, using a labeled set of images;
applying the initially trained text report machine learning model to a first set of unlabeled text reports with associated images to label the associated images;
selecting a first portion of labeled associated images;
re-training the image machine learning model using the selected first portion of labeled associated images;
applying the initially trained image machine learning model to a first set of unlabeled images with associated text reports to label the associated text reports;
selecting a first portion of labeled associated text reports; and
re-training the text report machine learning model using the selected first portion of labeled associated text reports.
10. The method of claim 9 , further comprising,
selecting a portion of labeled associated images including selecting associated images with text report machine learning model outputs having a confidence level above a first confidence threshold or
selecting a portion of labeled associated text reports including selecting associated text reports with image machine learning model outputs having a confidence level above a second confidence threshold.
11. The system of claim 9 , wherein
selecting a portion of labeled associated images further includes selecting the N associated images with text report machine learning model outputs having the highest confidence levels, and
selecting a portion of labeled associated text reports further includes selecting the M associated text reports with image machine learning model outputs having the highest confidence levels.
12. The method of claim 9 , wherein
selecting a portion of labeled associated images includes selecting associated images with text report machine learning model outputs having a confidence level above a first confidence threshold,
when there are more than N selected associated images, further selecting N associated Images with text reports that have the highest confidence levels,
selecting a portion of labeled associated text reports includes selecting associated text reports with image machine learning model outputs having a confidence level above a second confidence threshold, and
when there are more than M selected associated text reports, further selecting M associated text reports with images that have the highest confidence levels.
13. The method of claim 9 , wherein the outputs of the text report machine learning model and image machine learning model indicated the same set of classification concepts.
14. The method of claim 9 , wherein
the image machine learning model is re-trained until the initially trained text report machine learning model is applied to all the unlabeled text reports with associated images, and
the text report machine learning model is re-trained until the initially trained image machine learning model is applied to all the unlabeled images with associated text reports.
15. The method of claim 9 , wherein
the image machine learning model is re-trained until the image machine learning model performance on validation set of input image data does not improve, and
the text report machine learning model is re-trained until the text report machine learning model performance on a validation set of input text report data does not improve.
16. The method of claim 9 , further comprising:
applying the retrained text report machine learning model to a second set of unlabeled text reports with associated images to label the associated images;
selecting a second portion of labeled associated images;
re-training the retrained image machine learning model using the selected second portion of labeled associated images;
applying the retrained image machine learning model to a second set of unlabeled images with associated text reports to label the associated text reports;
selecting a second portion of labeled associated text reports; and
re-training the retrained text report machine learning model using the selected second portion of labeled associated text reports.
17. A non-transitory computer readable medium, comprising instructions executable by a processor to carry out the method of:
initially training a text report machine learning model, using a labeled set of text reports including text pre-processing the text report and extracting features from the pre-processed text report, wherein the extracted features are Input into the text report machine learning model;
initially training an image machine learning model, using a labeled set of images;
applying the initially trained text report machine learning model to a first set of unlabeled text reports with associated images to label the associated images;
selecting a first portion of labeled associated images;
re-training the image machine learning model using the selected first portion of labeled associated images;
applying the initially trained image machine learning model to a first set of unlabeled images with associated text reports to label the associated text reports;
selecting a first portion of labeled associated text reports; and
re-training the text report machine learning model using the selected first portion of labeled associated text reports.
18. The non-transitory computer readable medium of claim 17 , wherein
the image machine learning model is re-trained until the image machine learning model performance on validation set of input image data does not improve or the text report machine learning model is re-trained until the text report machine learning model performance on a validation set of input text report data does not improve.
19. The non-transitory computer readable medium of claim 17 , further comprising,
selecting a portion of labeled associated images including selecting associated images with text report machine learning model outputs having a confidence level above a first confidence threshold, or
selecting a portion of labeled associated text reports including selecting associated text reports with image machine learning model outputs having a confidence level above a second confidence threshold.
20. The non-transitory computer readable medium of claim 17 , wherein the text report machine learning model is one of a conditional random field (CRF) classifier, bidirectional long short-term memory (BILSTM) network, and BILSTM-CRF and the image machine learning model is one of a convolutional neural network (CNN) and class activation mapping model.