Systems and methods for directly predicting cancer patient survival based on histopathology images
One example method includes obtaining one or more histopathology images of a sample from a cancer patient; selecting a plurality of tissue image patches from the one or more histopathology images; determining, by a deep learning system comprising a plurality of trained machine learning (“ML”) models, a plurality of image features for the plurality of tissue image patch, wherein each tissue image patch is analyzed by one of the trained ML models; determining, by the deep learning system, probabilities of patient survival based on the determined plurality of image features; and generating, by the deep learning system, a prediction of patient survival based on the determined probabilities.
1 . A method comprising:
obtaining one or more histopathology images of a sample from a cancer patient;
selecting a plurality of tissue image patches from the one or more histopathology images;
determining, by a deep learning system comprising a plurality of trained machine learning (“ML”) models comprising a plurality of trained convolutional neural networks (“CNNs”) having convolutional layers having shared weights and having outputs connected to an average pool, the average pool connected to a fully connected layer, a plurality of image features for the plurality of tissue image patches, wherein each tissue image patch is analyzed by one of the trained ML models;
determining, by the deep learning system, probabilities of patient survival based on the determined plurality of image features; and
generating, by the deep learning system, a prediction of patient survival based on the determined probabilities.
2 . The method of claim 1 , wherein the plurality of tissue image patches are randomly selected.
3 . The method of claim 1 , wherein the plurality of tissue image patches are uniformly sized.
4 . The method of claim 1 , wherein the trained CNNs were trained to predict prognosis from a set of training images in accordance with a survival loss function as a probability distribution over discretized survival times.
5 . The method of claim 4 , wherein the deep learning system is trained without requiring the leveraging of expert annotations or known features of interest in the training images.
6 . The method of claim 4 , wherein the survival loss function comprises censored cross-entropy function.
7 . The method of claim 4 , further comprising making a prediction of a molecular characteristic in a tissue sample.
8 . The method of claim 7 , wherein the molecular characteristic comprises a genetic mutation.
9 . The method of claim 1 , wherein the deep learning system is tuned with one or more hyperparameters comprising fixation types, patch size, patch set size, magnification, number of layers, base depth, L2 regularization weight, initial learning rate, thresholds, or training data sets.
10 . A weakly supervised deep learning system for prediction of prognosis of a cancer patient, comprising:
a non-transitory computer-readable medium; and
a processor communicatively coupled to the non-transitory computer-readable medium, the processor configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:
obtain one or more histopathology images of a sample from a cancer patient;
randomly select a plurality of tissue image patches from the one or more histopathology images;
determine, by a deep learning system comprising a plurality of trained machine learning (“ML”) models comprising a plurality of trained convolutional neural networks (“CNNs”) having convolutional layers having shared weights and having outputs connected to an average pool, the average pool connected to a fully connected laver, a plurality of image features for the plurality of tissue image patches, wherein each tissue image patch is analyzed by one of the trained ML models;
determine, by the deep learning system, probabilities of patient survival based on the determined plurality of image features; and
generate, by the deep learning system, a prediction of patient survival based on the determined probabilities.
11 . The weakly supervised deep learning system of claim 10 , wherein the plurality of tissue image patches are uniformly sized.
12 . The weakly supervised deep learning system of claim 10 , wherein the plurality of tissue image patches are randomly selected.
13 . The weakly supervised deep learning system of claim 10 , wherein the trained CNNs were trained to predict prognosis from a set of training images in accordance with a survival loss function as a probability distribution over discretized survival times.
14 . The weakly supervised deep learning system of claim 13 , wherein the deep learning system is trained without requiring the leveraging of expert annotations or known features of interest in the training images.
15 . The weakly supervised deep learning system of claim 13 , wherein the survival loss function comprises censored cross-entropy function.
16 . The weakly supervised deep learning system of claim 13 , wherein the system is further trained to make a prediction of a molecular characteristic in a tissue sample.
17 . The weakly supervised deep learning system of claim 16 , wherein the molecular characteristic comprises a genetic mutation.
18 . The weakly supervised deep learning system of claim 10 , wherein the deep learning architecture is tuned with one or more hyperparameters comprising fixation types, patch size, patch set size, magnification, number of layers, base depth, L2 regularization weight, initial learning rate, thresholds, or training data sets.
19 . A weakly supervised deep learning system for prediction of prognosis of a cancer patient, comprising:
multiple convolutional neural networks modules having convolutional layers with shared weights, wherein each of the multiple convolutional neural networks accepts as input one cropped tissue image patch randomly selected from a histopathology image, an average pool receiving the output of each of the convolutional neural networks and a fully connected layer, wherein each of the multiple convolutional neural networks extracting features from each patch, the patch level features averaged on a per-channel basis in the average pool and fed to the fully connected layer,
the multiple convolutional neural networks trained from training images comprising at least one histopathology image from each of a multitude of cancer patients to predict prognosis in accordance with a survival loss function as a probability distribution over discretized survival times, and
wherein the deep learning system is trained without requiring the leveraging of expert annotations or known features of interest in the training images.
20 . The weakly supervised deep learning system of claim 19 , wherein the survival loss function comprises a censored cross-entropy function.
21 . The weakly supervised deep learning system of claim 19 , wherein the cancer patients have prostate cancer, breast cancer, colon cancer, head and neck squamous cell carcinoma, or liver hepatocellular carcinoma.
22 . The weakly supervised deep learning system of claim 19 , wherein the system is further trained to make a prediction of a molecular characteristic in a tissue sample.
23 . The weakly supervised deep learning system of claim 22 , wherein the molecular characteristic comprises a genetic mutation.
24 . The weakly supervised deep learning system of claim 19 , wherein the multiple convolutional neural networks are trained from at least one histopathology image from each of a multitude of cancer patients having a single type of cancer.
25 . A system comprising a plurality of the weakly supervised deep learning system for prediction of prognosis of a cancer patient of claim 19 , wherein each of the plurality of weakly supervised deep learning system is trained from the at least one histopathology image from each of a multitude of cancer patients having a single type of cancer, wherein the single type of cancer may be any of breast cancer, prostate cancer, colon cancer, head and neck squamous cell carcinoma, or liver hepatocellular carcinoma.