Systems and methods to process electronic images for continuous biomarker prediction
Systems and methods are disclosed for processing digital images to predict at least one continuous value comprising receiving one or more digital medical images, determining whether the one or more digital medical images includes at least one salient region, upon determining that the one or more digital medical images includes the at least one salient region, predicting, by a trained machine learning system, at least one continuous value corresponding to the at least one salient region, and outputting the at least one continuous value to an electronic storage device and/or display.
1 . A method comprising:
obtaining one or more histopathology images of a sample from a cancer patient, the one or more histopathology images includes pixels and/or voxels;
selecting a plurality of tissue image patches from the one or more histopathology images;
appending spatial information to pixels and/or voxels of the one or more histopathology images, wherein the spatial information is appended by concatenating coordinates of each pixel and/or voxel to each pixel and/or voxel;
determining, by a deep learning system comprising a plurality of trained machine learning (“ML”) models, a plurality of image features for the plurality of tissue image patch, wherein each tissue image patch is analyzed by one of the trained ML models;
determining, by the deep learning system, one or more continuous score based on the determined plurality of image features and based on analyzing the pixels and/or voxels with the appended spatial information, wherein determining the one or more continuous scores based on analyzing the pixels and/or voxels with appended spatial information comprises:
incorporating a predicted genomic expression, a predicted number of cells having a particular cell type, a predicted number of cells having a particular cell sub-type, a predicted protein expression, a predicted measurement of physical size, or a combination thereof, and
incorporating a location of one or more cells, a spatial distribution of predicted genomic expression, a spatial distribution of predicted protein expression, or a combination thereof; and
generating, by the deep learning system, a predicted severity of cancer based on the one or more continuous scores.
2 . The method of claim 1 , wherein the severity of cancer corresponds to a percentage chance of death.
3 . The method of claim 1 , wherein the plurality of tissue image patches are uniformly sized.
4 . The method of claim 1 , wherein the plurality of ML models is a plurality of trained convolutional neural networks (“CNNs”) with shared weights, wherein the trained CNNs were trained to predict prognosis from a set of training images.
5 . The method of claim 4 , wherein the deep learning system is trained without requiring leveraging of expert annotations or known features of interest in the training images.
6 . The method of claim 4 , further comprising making a prediction of a molecular characteristic in a tissue sample.
7 . The method of claim 6 , wherein the molecular characteristic comprises a genetic mutation.
8 . The method of claim 1 , wherein the continuous score represents years of expected survival for the cancer patient.
9 . The method of claim 1 , wherein the deep learning system is tuned with one or more hyperparameters comprising fixation types, patch size, patch set size, magnification, number of layers, base depth, L2 regularization weight, initial learning rate, thresholds, or training data sets.
10 . A weakly supervised deep learning system for prediction of prognosis of a cancer patient, comprising:
a non-transitory computer-readable medium; and
a processor communicatively coupled to the non-transitory computer-readable medium, the processor configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:
obtain one or more histopathology images of a sample from a cancer patient, the one or more histopathology images includes pixels and/or voxels;
select a plurality of tissue image patches from the one or more histopathology images;
append spatial information to pixels and/or voxels of the one or more histopathology images, wherein the spatial information is appended by concatenating coordinates of each pixel and/or voxel to each pixel and/or voxel;
determine, by a deep learning system comprising a plurality of trained machine learning (“ML”) models, a plurality of image features for the plurality of tissue image patch, wherein each tissue image patch is analyzed by one of the trained ML models;
determine, by the deep learning system, one or more continuous scores based on the determined plurality of image features and based on analysis of the pixels and/or voxels with the appended spatial information, wherein determining the one or more continuous scores based on analyzing the pixels and/or voxels with appended spatial information comprises:
incorporating a predicted genomic expression, a predicted number of cells having a particular cell type, a predicted number of cells having a particular cell sub-type, a predicted protein expression, a predicted measurement of physical size, or a combination thereof, and
incorporating a location of one or more cells, a spatial distribution of predicted genomic expression, a spatial distribution of predicted protein expression, or a combination thereof; and
generate, by the deep learning system, a predicted severity of cancer based on the one or more continuous scores.
11 . The weakly supervised deep learning system of claim 10 , wherein the severity of cancer corresponds to a percentage chance of death.
12 . The weakly supervised deep learning system of claim 10 , wherein the plurality of tissue image patches are randomly selected.
13 . The weakly supervised deep learning system of claim 10 , wherein the continuous score represents years of expected survival for the cancer patient.
14 . The weakly supervised deep learning system of claim 10 , wherein the plurality of ML models is a plurality of trained convolutional neural networks (“CNNs”), wherein the trained CNNs were trained to predict prognosis from a set of training images.
15 . The weakly supervised deep learning system of claim 13 , wherein the deep learning system is trained without requiring leveraging of expert annotations or known features of interest in the training images.
16 . The weakly supervised deep learning system of claim 13 , wherein the system is further trained to make a prediction of a molecular characteristic in a tissue sample.
17 . The weakly supervised deep learning system of claim 16 , wherein the molecular characteristic comprises a genetic mutation.
18 . The weakly supervised deep learning system of claim 10 , wherein the deep learning system is tuned with one or more hyperparameters comprising fixation types, patch size, patch set size, magnification, number of layers, base depth, L2 regularization weight, initial learning rate, thresholds, or training data sets.
19 . A weakly supervised deep learning system for prediction of a continuous score of a cancer patient, comprising:
multiple convolutional neural networks wherein one or more of the multiple convolutional neural networks is configured to extract features from one or more patches of a received histopathology image,
the multiple convolutional neural networks trained from training images comprising at least one histopathology image from each of a multitude of cancer patients to predict a continuous score, the wherein the multiple convolutional neural networks are trained by:
extracting at least one diagnostic feature vector/tensor from the one or more patches,
appending a spatial coordinate of each pixel and/or voxel of the one or more patches to a corresponding at least one diagnostic feature vector/tensor; and
based on the at least one diagnostic feature vector/tensor and analyzing the pixels and/or voxels with appended spatial information, determining the continuous score, wherein determining the continuous score based on analyzing the pixels and/or voxels with appended spatial information comprises:
incorporating a predicted genomic expression, a predicted number of cells having a particular cell type, a predicted number of cells having a particular cell sub-type, a predicted protein expression, a predicted measurement of physical size, or a combination thereof, and
incorporating a location of one or more cells, a spatial distribution of predicted genomic expression, a spatial distribution of predicted protein expression, or a combination thereof;
wherein the continuous score being based on a predicted severity of cancer from the at least one histopathology image, and
wherein the deep learning system is trained without requiring leveraging of expert annotations or known features of interest in the training images.
20 . The weakly supervised deep learning system of claim 19 , wherein the continuous score represents years of expected survival for the cancer.