Systems and methods to process electronic images for continuous biomarker prediction
Systems and methods are disclosed for processing digital images to predict at least one continuous value comprising receiving one or more digital medical images, determining whether the one or more digital medical images includes at least one salient region, upon determining that the one or more digital medical images includes the at least one salient region, predicting, by a trained machine learning system, at least one continuous value corresponding to the at least one salient region, and outputting the at least one continuous value to an electronic storage device and/or display.
1. A computer-implemented method of determining a continuous value prediction for one or more digital medical images, the method comprising:
receiving one or more digital medical images of a sample associated with a patient, the one or more digital medical images comprising pixels and/or voxels;
appending spatial information to the pixels and/or voxels of the one or more digital medical images, wherein the spatial information is appended by concatenating coordinates of each pixel and/or voxel to each pixel and/or voxel;
determining, by providing the appended digital medical images to a trained machine learning system, a continuous value prediction of a level of one or more biomarkers, the trained machine learning system having been trained directly using a plurality of medical images and a continuous score prediction loss function; and
providing the continuous value prediction for output to a display and/or storage,
wherein determining a continuous value prediction based on analyzing the pixels and/or voxels with appended spatial information comprises:
incorporating a predicted genomic expression, a predicted number of cells having a particular cell type, a predicted number of cells having a particular cell sub-type, a predicted protein expression, a predicted measurement of physical size, or a combination thereof; and
incorporating a location of one or more cells, a spatial distribution of predicted genomic expression, a spatial distribution of predicted protein expression, or a combination thereof.
2. The computer-implemented method of claim 1 , wherein the spatial information is appended throughout processing.
3. The computer-implemented method of claim 1 , wherein the spatial information is appended by the trained machine learning system, the trained machine learning system passively self-selecting regions of the one or more digital medical images to process.
4. The computer-implemented method of claim 1 , wherein a salient region detection module provides saliency data for each pixel and/or voxel and the trained machine learning system determines a continuous value prediction of a level of one or more biomarkers based on the saliency data.
5. The computer-implemented method of claim 4 , wherein the salient region detection module breaks each digital medical image into at least one region by at least one of creating tiles of the image, creating segmentations based on edge or contrast, creating segmentations based on color differences, creating segmentations based on energy minimization, using supervised determination by the trained machine learning system, and using EdgeBoxes.
6. The computer-implemented method of claim 1 , further comprising a visual indicator to alert a user to the level of the one or more continuous values.
7. The computer-implemented method of claim 1 , wherein the trained machine learning system outputs a continuous homologous recombination deficiency score based on one or more inputted pathology specimens.
8. The computer-implemented method of claim 1 , wherein the trained machine learning system outputs a continuous tumor mutation burden score based on one or more inputted pathology specimens.
9. The computer-implemented method of claim 1 , wherein the trained machine learning system outputs a count for each cell type of interest based on one or more inputted pathology specimens.
10. The computer-implemented method of claim 9 , wherein the count for each cell type of interest may be predicted using regression or counting loss functions.
11. The computer-implemented method of claim 1 , wherein the trained machine learning system outputs a continuous HER2 expression score based on one or more inputted pathology specimens.
12. The computer-implemented method of claim 1 , wherein the trained machine learning system outputs a continuous gene expression value based on one or more inputted pathology specimens.
13. The computer-implemented method of claim 1 , wherein the trained machine learning system outputs a Neurotrophic Receptor Tyrosine Kinase gene fusion score based on one or more inputted pathology specimens.
14. The computer-implemented method of claim 1 , wherein the trained machine learning system outputs a continuous microsatellite instability score based on one or more inputted pathology specimens, the continuous microsatellite instability score being derived via DNA sequencing.