Deep learning method for predicting patient response to a therapy
A method for indicating how a cancer patient will respond to a predetermined therapy relies on spatial statistical analysis of classes of cell centers in a digital image of tissue of the cancer patient. The cell centers are detected in the image of stained tissue of the cancer patient. For each cell center, an image patch that includes the cell center is extracted from the image. A feature vector is generated based on each image patch using a convolutional neural network. A class is assigned to each cell center based on the feature vector associated with each cell center. A score is computed for the image of tissue by performing spatial statistical analysis based on classes of the cell centers. The score indicates how the cancer patient will respond to the predetermined therapy. The predetermined therapy is recommended to the patient if the score is larger than a predetermined threshold.
1. A method comprising:
detecting cell centers on a digital image of tissue of a cancer patient, wherein the tissue has been stained;
for each cell center, extracting an image patch that includes the cell center;
generating a feature vector based on each image patch using a convolutional neural network;
assigning a class to each cell center based on the feature vector associated with each image patch that includes the cell center; and
computing a score for the digital image of tissue based on how the cell centers assigned to each of the classes of cell centers are spatially distributed in the digital image and based on how many cell centers assigned to a first class are located closer than a predetermined distance to a predetermined number of cell centers assigned to a second class, wherein the score is indicative of how the cancer patient will respond to a predetermined therapy.
2. The method of claim 1 , further comprising:
recommending the predetermined therapy if the score is larger than a predetermined threshold.
3. The method of claim 1 , wherein the score is indicative of a survival probability of the cancer patient after receiving the predetermined therapy.
4. The method of claim 1 , further comprising:
determining how the cell centers assigned to each of the classes of cell centers are spatially distributed in the digital image by counting how many cell centers are assigned to each of the classes of cell centers.
5. The method of claim 1 , wherein the score for the digital image of tissue of the cancer patient is computed by comparing how the cell centers assigned to each of the classes of cell centers are spatially distributed in the digital image of tissue of the cancer patient to how cell centers of classes of cell centers are spatially distributed in digital images of tissue from a plurality of cancer patients whose responses to the predetermined therapy are known.
6. The method of claim 1 , wherein the classes of the cell centers are defined using k-means clustering of feature vectors of cell centers detected in a plurality of digital images of tissue from a plurality of cancer patients.
7. The method of claim 1 , wherein the convolutional neural network is a discriminator part of an information maximizing generative adversarial network (infoGAN), and wherein the classes of the cell centers are defined by performing an arg-max operation on output values of predefined network nodes of the convolutional neural network.
8. The method of claim 1 , wherein the convolutional neural network is a convolutional neural network selected from the group consisting of: a generative adversarial network (GAN), an information maximizing generative adversarial network (infoGAN), a boundary equilibrium generative adversarial network (BeGAN), and an autoencoder convolutional neural network.
9. The method of claim 1 , wherein the tissue has been stained with a stain that stains cellular nuclei.
10. The method of claim 1 , wherein the tissue has been stained using an antibody specific to a protein selected from the group consisting of: HER1, HER2, HER3, KI67, PD L1, PD-L2, CD3, CD4, CD8, CD20, and CD73.
11. The method of claim 1 , wherein the tissue has been immunohistochemically stained using hematoxylin and eosin.
12. The method of claim 1 , wherein the extracting the image patch involves extracting a first image patch that includes the cell center and then extracting a second image patch that is smaller than the first image patch and centered on the cell center.
13. A method comprising:
detecting cell centers on a digital image of tissue of a cancer patient, wherein the tissue has been stained;
for each cell center, extracting an image patch that includes the cell center;
generating a feature vector based on each image patch using a convolutional neural network;
assigning a class to each cell center based on the feature vector associated with each image patch that includes the cell center; and
computing a score for the digital image of tissue based on how the cell centers assigned to each of the classes of cell centers are spatially distributed in the digital image and based on how many cell centers assigned to a first class are located closer than a predetermined distance to a predetermined number of cell centers assigned to a second class, wherein the score is indicative of a survival probability of the cancer patient.
14. The method of claim 13 , wherein the score for the digital image of tissue of the cancer patient is computed by comparing how the cell centers assigned to each of the classes of cell centers are spatially distributed in the digital image of tissue of the cancer patient to how cell centers of classes of cell centers are spatially distributed in digital images of tissue from a plurality of cancer patients whose survival probabilities are known.
15. The method of claim 13 , further comprising:
wherein determining how the cell centers assigned to each of the classes of cell centers are spatially distributed in the digital image is based on how many feature vectors are classified as belonging to a class of cell centers.
16. The method of claim 13 , wherein the convolutional neural network is a convolutional neural network selected from the group consisting of: a generative adversarial network (GAN), an information maximizing generative adversarial network (infoGAN), a boundary equilibrium generative adversarial network (BeGAN), and an autoencoder convolutional neural network.
17. The method of claim 13 , wherein the tissue has been immunohistochemically stained to mark membranes.
18. The method of claim 13 , wherein the extracting the image patch involves extracting a first image patch that includes the cell center and then extracting a second image patch that is larger than the first image patch and has a common center pixel with the first image patch.
19. The method of claim 13 , further comprising:
determining how the cell centers assigned to each of the classes of cell centers are spatially distributed in the digital image by determining how cell centers having similar feature vectors are spatially distributed in the digital image.
20. The method of claim 13 , wherein the score for the digital image of tissue of the cancer patient is computed by comparing how cell centers having feature vectors that share a common feature are spatially distributed in the digital image to how cell centers having feature vectors that do not share the common feature are spatially distributed in the digital image.