Systems, methods, and apparatuses for implementing self-supervised domain-adaptive pre-training via a transformer for use with medical image classification
Described herein are systems, methods, and apparatuses for implementing self-supervised domain-adaptive pre-training via a transformer for use with medical image classification in the context of medical image analysis. An exemplary system includes means for receiving a first set of training data having non-medical photographic images; receiving a second set of training data with medical images; pre-training an AI model on the first set of training data with the non-medical photographic images; performing domain-adaptive pre-training of the AI model via self-supervised learning operations using the second set of training data having the medical images; generating a trained domain-adapted AI model by fine-tuning the AI model against the targeted medical diagnosis task using the second set of training data having the medical images; outputting the trained domain-adapted AI model; and executing the trained domain-adapted AI model to generate a predicted medical diagnosis from an input image not present within the training data.
1. A system comprising:
a memory to store instructions;
a processor to execute the instructions stored in the memory;
wherein the system is specially configured to execute instructions via the processor for performing the following operations:
receiving at the system, a first set of training data which includes photographic images unrelated to a targeted medical diagnosis task;
receiving at the system, a second set of training data which includes a plurality of medical images derived from multiple distinct sources, wherein the plurality of medical images are configured with multiple inconsistent annotation and classification data;
executing instructions via the processor of the system for pre-training an AI model on the first set of training data which includes the photographic images by learning image classification from the photographic images within the first set of training data;
executing instructions via the processor of the system for performing domain-adaptive pre-training of the AI model via self-supervised learning operations, wherein the AI model previously trained on the first set of training data applies domain-adaptive learning to scale up data utilization of the second set of training data within the AI model's learned image classifications;
generating a trained domain-adapted AI model by fine-tuning the AI model against the targeted medical diagnosis task using the second set of training data which includes the plurality of medical images;
outputting the trained domain-adapted AI model; and
executing the trained domain-adapted AI model to generate a predicted medical diagnosis from an input image which forms no part of the first or second sets of training data.
2. The system of claim 1 :
wherein pre-training the AI model on the first set of training data generates a trained AI model; and
wherein performing domain-adaptive pre-training of the AI model via self-supervised learning operations generates a trained domain-adapted AI model by fine-tuning the trained AI model against the targeted medical diagnosis task using the second set of training data which includes the plurality of medical images.
3. The system of claim 1 , further comprising:
performing in-domain medical transfer learning for via the trained domain-adapted AI model by prioritizing all in-domain medical transfer learnings derived from the second set of training data which includes the plurality of medical images over any out-of-domain learnings derived from the pre-training of the AI model on the first set of training data generates a trained AI model.
4. The system of claim 3 , wherein prioritizing all in-domain medical transfer learnings is configured to improve performance of the trained domain-adapted AI model by reducing domain disparities between a source domain corresponding to the first set of training data and a target domain corresponding to the second set of training data which includes the plurality of medical images.
5. The system of claim 1 , wherein receiving the first set of training data which includes photographic images unrelated to a targeted medical diagnosis task comprises receiving photographic images corresponding to a first domain type which lacks medical imaging data.
6. The system of claim 1 , wherein receiving the second set of training data which includes a plurality of medical images comprises receiving medical imaging data corresponding to a second domain type having at least a sub-set of the plurality of medical images correlated to the targeted medical diagnosis task.
7. The system of claim 1 , wherein receiving the first set of training data which includes photographic images unrelated to a targeted medical diagnosis task comprises receiving a set of non-domain specific photographic images lacking any images correlated to the targeted medical diagnosis task.
8. The system of claim 1 , further comprising:
receiving, at the system, multiple medical imaging training data sets from multiple distinct sources;
aggregating the multiple medical imaging training data sets into a single aggregated medical imaging dataset; and
wherein, receiving at the system, the second set of training data which includes the plurality of medical images derived from multiple distinct sources comprises specifying the single aggregated medical imaging dataset as the second set of training data for performing the domain-adaptive pre-training of the AI model.
9. The system of claim 1 , wherein receiving the second set of training data which includes the plurality of medical images derived from multiple distinct sources includes training data incompatible with supervised learning for the AI model.
10. The system of claim 1 , wherein the plurality of medical images received with the second set of training data are configured with multiple inconsistent annotation and classification data including at least two or more of:
inconsistent annotations across a sub-set of the plurality of medical images for the targeted medical diagnosis task;
inconsistent annotations across a sub-set of the plurality of medical images for a common disease condition represented within the plurality of medical images;
inconsistent annotations across a sub-set of the plurality of medical images for a common human anatomical feature classified within the plurality of medical images;
inconsistent global level image annotations identifying disease conditions within an image and local level boxed-lesion labels identifying disease conditions within bounding boxes present within the plurality of medical images;
inconsistent use of expert annotations for the plurality of medical images with at least a first portion of the plurality of medical images including expert annotations and at least a second portion of the plurality of medical images lacking any expert annotations; and
inconsistent use of radiological reports associated with the plurality of medical images with at least a first portion of the plurality of medical images having radiological reports associated with them and at least a second portion of the plurality of medical images lacking any associated radiological reports.
11. The system of claim 1 , wherein performing the domain-adaptive pre-training of the AI model via the self-supervised learning operations,
bridges a domain gap between photographic images from the first set of training data representing a first domain and medical images from the second set of training data representing a second domain;
wherein the self-supervised learning operations operate without requiring expert annotation of the medical images from the second set of training data which exhibit inconsistent or missing labeling and inconsistent or missing expert annotations.
12. The system of claim 1 , wherein performing the domain-adaptive pre-training of the AI model via self-supervised learning operations, comprises:
applying continual pre-training on a large-scale domain-specific dataset represented within the second set of training data having the plurality of medical images derived from the multiple distinct sources;
wherein the continual pre-training is performed via self-supervised Masked Image Modeling (MIM) learning by the AI model; and
wherein the self-supervised Masked Image Modeling (MIM) learning by the AI model mitigates over-fitting in the targeted medical diagnosis task.
13. A computer-implemented method performed by a system having at least a processor and a memory therein for executing instructions, wherein the computer-implemented method comprises:
receiving at the system, a first set of training data which includes photographic images unrelated to a targeted medical diagnosis task;
receiving at the system, a second set of training data which includes a plurality of medical images derived from multiple distinct sources, wherein the plurality of medical images are configured with multiple inconsistent annotation and classification data;
executing instructions via the processor of the system for pre-training an AI model on the first set of training data which includes the photographic images by learning image classification from the photographic images within the first set of training data;
executing instructions via the processor of the system for performing domain-adaptive pre-training of the AI model via self-supervised learning operations, wherein the AI model previously trained on the first set of training data applies domain-adaptive learning to scale up data utilization of the second set of training data within the AI model's learned image classifications;
generating a trained domain-adapted AI model by fine-tuning the AI model against the targeted medical diagnosis task using the second set of training data which includes the plurality of medical images;
outputting the trained domain-adapted AI model; and
executing the trained domain-adapted AI model to generate a predicted medical diagnosis from an input image which forms no part of the first or second sets of training data.
14. The computer-implemented method of claim 13 :
wherein pre-training the AI model on the first set of training data generates a trained AI model; and
wherein performing domain-adaptive pre-training of the AI model via self-supervised learning operations generates a trained domain-adapted AI model by fine-tuning the trained AI model against the targeted medical diagnosis task using the second set of training data which includes the plurality of medical images.
15. The computer-implemented method of claim 13 , further comprising:
performing in-domain medical transfer learning for via the trained domain-adapted AI model by prioritizing all in-domain medical transfer learnings derived from the second set of training data which includes the plurality of medical images over any out-of-domain learnings derived from the pre-training of the AI model on the first set of training data generates a trained AI model; and
wherein prioritizing all in-domain medical transfer learnings is configured to improve performance of the trained domain-adapted AI model by reducing domain disparities between a source domain corresponding to the first set of training data and a target domain corresponding to the second set of training data which includes the plurality of medical images.
16. The computer-implemented method of claim 13 , wherein receiving the first set of training data which includes photographic images unrelated to a targeted medical diagnosis task comprises receiving photographic images corresponding to a first domain type which lacks medical imaging data; and
wherein receiving the second set of training data which includes a plurality of medical images comprises receiving medical imaging data corresponding to a second domain type having at least a sub-set of the plurality of medical images correlated to the targeted medical diagnosis task.
17. The computer-implemented method of claim 13 , wherein receiving the first set of training data which includes photographic images unrelated to a targeted medical diagnosis task comprises receiving a set of non-domain specific photographic images lacking any images correlated to the targeted medical diagnosis task.
18. The computer-implemented method of claim 13 , further comprising:
receiving, at the system, multiple medical imaging training data sets from multiple distinct sources;
aggregating the multiple medical imaging training data sets into a single aggregated medical imaging dataset; and
wherein, receiving at the system, the second set of training data which includes the plurality of medical images derived from multiple distinct sources comprises specifying the single aggregated medical imaging dataset as the second set of training data for performing the domain-adaptive pre-training of the AI model.
19. The computer-implemented method of claim 13 , wherein the plurality of medical images received with the second set of training data are configured with multiple inconsistent annotation and classification data including at least two or more of:
inconsistent annotations across a sub-set of the plurality of medical images for the targeted medical diagnosis task;
inconsistent annotations across a sub-set of the plurality of medical images for a common disease condition represented within the plurality of medical images;
inconsistent annotations across a sub-set of the plurality of medical images for a common human anatomical feature classified within the plurality of medical images;
inconsistent global level image annotations identifying disease conditions within an image and local level boxed-lesion labels identifying disease conditions within bounding boxes present within the plurality of medical images;
inconsistent use of expert annotations for the plurality of medical images with at least a first portion of the plurality of medical images including expert annotations and at least a second portion of the plurality of medical images lacking any expert annotations; and
inconsistent use of radiological reports associated with the plurality of medical images with at least a first portion of the plurality of medical images having radiological reports associated with them and at least a second portion of the plurality of medical images lacking any associated radiological reports.
20. A system comprising:
a memory to store instructions;
a processor to execute the instructions stored in the memory;
wherein the system is specially configured to execute instructions for systematically benchmarking vision transformers for use with chest x-ray classification, by performing operations including:
receiving first user input specifying multiple vision transformers;
receiving second user input specifying multiple training image datasets;
generating a list of all possible combinations of the multiple vision transformers specified and the multiple training image datasets specified according to the first and second user inputs;
retrieving a pre-trained base model for each of the multiple vision transformers specified and storing the pre-trained base models retrieved in the memory of the system for local execution;
retrieving the multiple training image datasets specified and storing the multiple training image datasets locally at the system;
initializing each of the pre-trained base models stored in memory using a standardized ImageNet dataset with randomized initialization weights to generate initialized vision transformer models corresponding to each of the pre-trained base models stored in memory;
executing Self-Supervised Learning (SSL) against each of the previously initialized vision transformer models using each of the multiple training image datasets corresponding to the list of all possible combinations previously generated to produce as output multiple SSL trained vision transformer models corresponding to the list of all possible combinations previously generated;
executing, via the processor of the system, each of the multiple SSL trained vision transformer models to generate image classification results as output from each of the multiple SSL trained vision transformer models; and
outputting from the system, a ranking of the image classification results generated as the output from each of the multiple SSL trained vision transformer models according to an area under the curve percentage calculation.