Systems and methods for mesothelioma feature detection and enhanced prognosis or response to treatment
A method and apparatus of a device that classifies a mesothelioma image is described. In an exemplary embodiment, the device segments the mesothelioma image into a region of interest that includes information useful for classification, and a background region, by applying a first convolutional neural network. In addition, the device tiles the region of interest into a set of tiles. For each tile, the device extracts a feature vector of that tile by applying a second convolutional neural network, where the features of the feature vectors represent local descriptors of the tile. Furthermore, the device processes the extracted feature vectors of the set of tiles to classify the image.
1. A method for determining the prognosis of a subject known or suspected to have mesothelioma, comprising:
accessing a biopsy image obtained from the subject;
extracting a plurality of feature vectors of the biopsy image by applying a first convolutional neural network, wherein each of the features of the plurality of feature vectors represents local descriptors of the biopsy image and presence or absence of one or more mesothelioma features associated with survival duration;
classifying the biopsy image using at least the plurality of feature vectors and a classification model, wherein the classification model is trained using a training set of known mesothelioma images and each of the known mesothelioma images has a corresponding label with survival duration information; and
determining the prognosis of the subject based on at least the classification of the biopsy image, wherein the prognosis is an indication of a survival duration of the subject.
2. The method of claim 1 , further comprising:
segmenting the biopsy image into a region of interest that includes information useful for classification and a background region by applying a second convolutional neural network.
3. The method of claim 2 , wherein the second convolutional neural network is a semantic segmentation deep learning network.
4. The method of claim 1 , further comprising:
tiling a region of interest of the biopsy image into a set of tiles, wherein each of the plurality of feature vectors corresponds to a tile from the set of tiles.
5. The method of claim 4 , wherein the tiling comprises:
applying a fixed tiling grid to at least the region of interest, wherein each of the set of tiles has a predetermined size.
6. The method of claim 4 , wherein the tiling comprises:
computing a score for each tile in the set of tiles using at least a convolutional 1D layer and the corresponding feature vector for that tile.
7. The method of claim 6 , wherein the classification comprises:
applying the classification model to a subset of tile scores to classify the biopsy image.
8. The method of claim 7 , wherein the classification model is a multi-layer perceptron with two connected layers.
9. The method of claim 6 , wherein the tiling further comprises:
determining the subset of tile scores by,
picking a highest set of tile scores, and
picking a lowest set of tile scores.
10. The method of claim 1 , wherein the first convolutional neural network is a ResNet50 neural network.
11. The method of claim 1 , wherein the extraction further comprises:
applying an autoencoder on the extracted plurality of feature vectors to reduce a dimensionality of the features of the plurality of feature vectors.
12. The method of claim 1 , wherein the training set of known mesothelioma images lacks local annotations of histopathological features.
13. The method of claim 1 , wherein the biopsy image lacks local annotations of histopathological features.
14. The method of claim 1 , wherein the known mesothelioma prognosis indications represent the duration of survival associated with the known mesothelioma images.
15. The method of claim 1 , wherein the prognosis of the subject is a risk score.
16. The method of claim 15 , wherein the risk score represents an estimated survival duration.
17. The method of claim 1 , wherein the biopsy image is selected from a digitized whole slide image (WSI) and a digitized image of a pathology section obtained from a biopsy stained with hematoxylin and eosin (H&E).
18. A method for generating a classification model for mesothelioma prognosis prediction, the method comprising:
receiving a training set of biopsy images, wherein each of the training set of biopsy images and each of the biopsy images has a label with survival duration information;
for each of the biopsy images in the training set of biopsy images,
extracting a plurality of feature vectors of the biopsy image by applying a first convolutional neural network, wherein each of the features of the plurality of feature vectors represents local descriptors of that biopsy image and presence or absence of one or more mesothelioma features associated with survival duration; and
training the classification model using at least the extracted feature vectors and the corresponding labels with survival duration information.
19. The method of claim 18 , further comprising:
validating the classification model using at least a validation set of biopsy images.
20. The method of claim 19 , wherein the validation comprises:
receiving the validation set of biopsy images, wherein each of the validation set of biopsy images has an associated known classification;
for each of the biopsy images in the validation set of biopsy images,
extracting a plurality of feature vectors of that biopsy image by applying a first convolutional neural network, wherein each of the features of the plurality of feature vectors represents local descriptors of that biopsy image,
generating a validation score set for that biopsy image using at least the plurality feature vectors, and
generating a classification for that biopsy image using at least the validation score set and the classification model; and
comparing the plurality of generated classifications with the associated known classifications.
21. The method of claim 20 , wherein the comparison of the plurality of generated classifications is performed using an area under the receiver operating characteristic curve (ROC-AUC) comparison, an area under the precision recall curve (PR-AUC) comparison, or a concordance index (c-index) comparison.
22. The method of claim 18 , wherein the classification model includes a multi-layer perceptron with two connected layers.
23. The method of claim 18 , wherein a biopsy image of the training set or validation set is one of a digitized whole slide image (WSI) and/or a digitized image of a pathology section obtained from a biopsy stained with hematoxylin and eosin (H&E).
24. The method of claim 18 , wherein each of the associated known classifications is a known mesothelioma survival value, a known mesothelioma subtype, and/or a known mesothelioma grade.
25. The method of claim 18 , wherein the first convolutional neural network is a ResNet50 neural network.
26. The method of claim 18 , wherein the extracting of the plurality of feature vectors comprises:
tiling a region of interest of the biopsy image into a set of tiles, wherein each of the plurality of feature vectors corresponds to a tile from the set of tiles.
27. The method of claim 26 , wherein the tiling comprises:
applying a fixed tiling grid to at least the region of interest, wherein each of the set of tiles has a predetermined size.
28. The method of claim 27 , wherein the training the classification model comprises:
computing a score for each tile in the set of tiles using at least a convolutional 1D layer and the corresponding feature vector for that tile.
29. The method of claim 18 , wherein for each of the biopsy images in the training set of biopsy images, the method further comprises:
segmenting that biopsy image into a region of interest that includes information useful for classification and a background region by applying a second convolutional neural network.
30. The method of claim 29 , wherein the second convolutional neural network is a semantic segmentation deep learning network.