SYSTEMS AND METHODS FOR DEEP LEARNING MODEL ANNOTATION USING SPECIALIZED IMAGING MODALITIES
In some aspects, a method, a system, or a non-transitory computer-readable storage medium are described for using a machine learning (ML) model to obtain annotations of a pathology slide image obtained in a first imaging modality, where the ML model is trained based in part on images obtained from a second imaging modality different from the first imaging modality. The first imaging modality is a conventional scanner for whole-slide images (WSI). The second imaging modality may include one or more of multispectral imaging (MSI), polarization imaging, quantitative phase imaging, or a combination thereof. The trained ML model can generate annotations that include more details with higher accuracy in comparison to annotating based on the WSI images alone.
1 . A method comprising:
using a machine learning (ML) model to obtain annotations of a pathology slide image obtained in a first imaging modality;
wherein the ML model is trained based in part on images obtained from a second imaging modality different from the first imaging modality.
2 . The method of claim 1 , wherein:
the first imaging modality is configured to image a slide based on light source of visible wavelengths and absorption of light by tissue.
3 . The method of claim 2 , wherein:
the second imaging modality comprises one or more of multispectral imaging (MSI), polarization imaging, quantitative phase imaging, or a combination thereof.
4 . The method of claim 3 , further comprising:
training the ML model, using a plurality of pairs of first image and second images;
wherein:
the first image in the pair is obtained from the first modality imaging of a first pathology slide; and
the second image in the pair is generated based on a second modality imaging of a second pathology slide corresponding to the first pathology slide.
5 . The method of claim 4 , wherein the second pathology slide and the first pathology slide are a same physical slide.
6 . The method of claim 4 , wherein:
the training further includes registering the first image and the second image in each of the pairs of first image and second image.
7 . The method of claim 6 , wherein the registering includes aligning the first image and the second image in each of the pairs.
8 . The method of claim 6 , wherein the second image in the pair is an annotation image comprising a plurality of objects each associated with a respective portion of the second image.
9 . The method of claim 8 , further comprising generating the annotation image by processing an image captured by the second modality imaging over a physical slide.
10 . The method of claim 8 , further comprising generating the annotation image based on a plurality of images captured by the second modality imaging over a physical slide.
11 . The method of claim 1 , further comprising generating HIFs from the annotations.
12 . The method of claim 11 , further comprising:
using a second ML to predict cell/tissue from the pathology slide image; and
generating the HIFs based additionally on the predicted cell/tissue.
13 . The method of claim 11 , further comprising predicting a disease based on the HIFs, using a statistical model.
14 . The method of claim 1 , wherein the annotations of the pathology slide image comprise heatmaps or labels of tissues/cells in the pathology slide image.
15 . A method comprising:
using a machine learning (ML) model to obtain annotations of a pathology slide image of a first type;
wherein the ML model is trained based in part on training pathology slide images of a second type different from the first type.
16 . The method of claim 15 , wherein:
the first type of image is obtained from a stained slide; and
the second type of image is a stain-invariant image obtained from a triplex slide.
17 . The method of claim 16 , wherein the second type of image is a phase image.