Multiple instance learner for tissue image classification
The method includes, for each of a plurality of tiles of an image, extracting a feature vector from the tile; providing a Multiple-Instance-Learning program configured to use a model for classifying any input image as a member of one out of at least two different classes based on feature vectors extracted from the tiles; for each of the tiles, computing a certainty value indicating the certainty of the model regarding the contribution of the tile's feature vector on the classification of the image; for each of the images, using, by the MIL-program, a certainty-value-based pooling function for aggregating the feature vectors of the image or predictive values computed from the feature vectors of the image into an aggregated predictive value as a function of the certainty values of the tiles; and classifying each of the images as a member of one of the classes based on the aggregated predictive value.
1 . A method for classifying tissue images, the method comprising:
receiving, by an image analysis system, a plurality of digital images, each of the digital images depicting a tissue sample of a patient;
splitting, by the image analysis system, each received image into a set of image tiles;
for each of the tiles, computing, by the image analysis system, a feature vector comprising image features extracted selectively from the tile;
providing a Multiple-Instance-Learning (MIL) program configured to use a model for classifying any input image as a member of one out of at least two different classes based on the feature vectors extracted from all tiles of the said input image, wherein the MIL-program is a neural network;
for each of the tiles, computing a certainty value, the certainty value being indicative of the certainty of the model regarding the contribution of the tile's feature vector on the classification of the image from which the tile was derived, wherein the certainty-value is computed using a dropout technique at test time of the model of the neural network;
for each of the images,
using, by the MIL-program, a certainty-value-based pooling function for aggregating the feature vectors extracted from the image into a global feature vector as a function of the certainty values of the tiles of the image, and computing an aggregated predictive value from the global feature vector; or
computing, by the MIL program, a predictive value from each of the feature vectors of the image and using, by the MIL-program, the certainty-value-based pooling function for aggregating the predictive values of the image into an aggregated predictive value as a function of the certainty values of the tiles of the image; and
classifying, by the MIL-program, each of the images as a member of one out of the at least two different classes based on the aggregated predictive value.
2 . The method of claim 1 , further comprising:
outputting, via a GUI, the classification result to a user; and/or
outputting the classification result to another application program.
3 . The method of claim 1 ,
wherein the MIL-program is a binary MIL-program,
wherein the at least two classes consist of a first class referred to as “positive class” and a second class referred to as “negative class”,
wherein any one of the images is classified into the “positive class” if the MIL model predicts for at least one of the tiles of this image that the feature vector of this tile comprises evidence for the “positive class”,
wherein any one of the images is classified into the “negative class” if the MIL model predicts for all the tiles of this image that their respective feature vectors do not comprises evidence for the “positive class”.
4 . The method of claim 1 , the certainty-value-based-pooling function being used at test time, the providing of the MIL-program comprising:
extracting feature vectors from a set of training tiles generated from a set of training images;
training the MIL-program on the feature vectors, using at training time the same certainty-value-based-pooling function as used at test time or using at training time another certainty-value-based-pooling function than the certainty-value-based-pooling function as used at test time,
wherein the certainty-value-based-pooling function used at training time is a certainty-value-based-max-pooling function or a certainty-value-based-mean-pooling function and wherein the certainty-value-based-pooling function used at test time is a certainty-value-based-max-pooling function.
5 . The method of claim 1 , wherein the certainty-value-based pooling function is a certainty-value-based-max-pooling function, wherein the using of the certainty-value-based pooling function comprises, for each of the images, a sub-method a), b, c) or d), respectively comprising:
a1) weighting the predictive value of each of the tiles with the certainty value computed for this tile, thereby obtaining a weighed predictive value;
a2) identifying the maximum of all weighted predictive values computed for all the tiles of the image; and
a3) using the maximum weighted predictive value as the aggregated predictive value; or
b) using the predictive value of the tile with the maximum certainty value as the aggregated predictive value; or
c1) weighting the feature vector of each of the tiles with the certainty value computed for this tile, thereby obtaining a weighed feature vector;
c2) identifying the maximum of all weighted feature vectors computed for all the tiles of the image; or
d) using the feature vector of the tile with the maximum certainty value as the global feature vector.
6 . The method of claim 1 , wherein the certainty-value-based pooling function is a certainty-value-based-mean-pooling function, wherein the using of the certainty-value-based pooling function comprises, for each of the images:
weighting the feature vector of each of the tiles with the certainty value computed for this tile, thereby obtaining a weighted feature vector; and computing the global feature vector as the mean of all the weighted feature vectors of the image; or
weighting the predictive value of each of the tiles with the certainty value computed for this tile, thereby obtaining a weighted predictive value; computing the mean of the weighted predictive values of the image; and using the computed mean as the aggregated predictive value.
7 . The method of claim 1 , wherein the certainty-value is further computed using the dropout technique at training of the model of the neural network.
8 . The method of claim 7 , wherein the certainty-value is computed as Monte-Carlo Dropout.
9 . The method of claim 1 , wherein the dropout technique and/or the certainty-value-based pooling function is used at test time but not at training time of the model.
10 . The method of claim 8 ,
wherein the neural network comprises one or more deactivated dropout layers,
wherein a deactivated dropout layer is a dropout layer activated at training time and deactivated at test time, the method comprising reactivating the one or more dropout layers at test time; or
wherein the neural network at training time is free of any dropout layer, the method comprising adding one or more dropout layers at test time to the neural network; and
wherein the computing of the certainty value for any one of the tiles at test time further comprises:
computing, for each of the tiles, multiple times a predictive value based on the feature vector extracted from the tile, wherein each time the predictive value is computed a different subset of nodes of the network is dropped by the one or more reactivated or added dropout layers; and
computing, for each of the tiles, the certainty value of the tile as a function of the variability of the multiple predictive values computed for the tile, wherein the larger the variability, the lower the certainty value.
11 . The method of claim 1 , the received digital images comprising:
digital images of tissue samples whose pixel intensity values correlate with the amount of a non-biomarker specific stain; and/or
digital images of tissue samples whose pixel intensity values correlate with the amount of a biomarker specific stain, the biomarker-specific stain adapted to selectively stain a biomarker contained in the tissue sample.
12 . The method of claim 1 , the providing of the MIL-program comprising training the model of the MIL-program, the training comprising one or more of:
providing a set of digital training images of tissue samples, each digital training image having assigned a class label being indicative of one of the at least two classes;
splitting each training image into training image tiles, each training tile having assigned the same class label as the digital training image from which the training tile was derived;
for each of the training tiles, computing, by the image analysis system, a training feature vector comprising image features extracted selectively from the said tile; and
repeatedly adapting the model of the MIL-program such that an error of a loss function is minimized, the error of the loss function indicating a difference of predicted class labels of the training tiles and the class labels actually assigned to the training tiles, the predicted class labels having been computed by the model based on the feature vector of the training tiles.
13 . The method of claim 1 , further comprising, for each of the received digital images:
weighting the predictive value of each of the tiles with the certainty value computed for this tile, thereby obtaining a weighted predictive value;
identifying, by the MIL-program, the one of the tiles of the image for which the highest weighted predictive value was computed;
for each of the other tiles of the image, computing a relevance indicator by comparing the weighted predictive value of the other tile with the highest weighted predictive value, wherein the relevance indicator is a numerical value that negatively correlates with the difference of the compared weighted predictive values;
computing a relevance heat map for the image as a function of the relevance indicator, the pixel color and/or pixel intensities of the relevance heat map being indicative of the relevance indicator computed for the tiles in the said image; and
displaying the relevance heat map on a GUI.
14 . The method of claim 1 , further comprising:
displaying the received images on a GUI of a screen, the images being grouped in the GUI into the at least two different classes in accordance with a result of the classification.
15 . The method of claim 1 , wherein each of the at least two classes is selected from a group comprising:
a patient being responsive to a particular drug;
a patient having developed metastases or a particular form of metastases;
a cancer patient showing a particular response to a particular therapy;
a cancer patient tissue showing a particular morphological state or microsatellite status;
a patient has developed adverse reaction to a particular drug;
a patient having a particular genetic attribute; and/or
a patient having a particular RNA expression profile.
16 . The method of claim 1 , wherein the certainty-value-based pooling function is a certainty-value based max-pooling, mean-pooling or an attention pooling function.
17 . The method of claim 1 , further comprising:
providing a trained attention-MLL having learned which tile-derived feature vectors are the most relevant for predicting class membership for a respective tile;
computing an attention weight for each of the tiles as a function of the feature vector of the respective tile by the attention-MLL, the attention weight being an indicator of the relevance of the respective tile's feature value in respect to a membership of the respective tile to a class; and
multiplying the attention weight of the respective tile with the respective tile's feature vector values for obtaining an attention-based feature vector with attention-weighted feature values for the respective tile; and wherein the method further comprises:
using the attention-based feature vector as the feature vector that is input to the MIL-program for computing the predictive value, the certainty value and/or the weighted predictive value of the respective tile, the predictive value, the certainty value and/or the weighted predictive value thereby being computed as attention-based predictive value, an attention-based certainty value and/or as an attention-based weighted predictive value; or
multiplying the attention weight of the respective tile with the respective tile's predictive value, the certainty value or with the respective tile's weighted predictive value computed by the MIL for obtaining an attention-based predictive value, an attention-based certainty value and/or an attention-based weighted predictive value.
18 . An image analysis system for classifying tissue images, comprising:
at least one processor;
a volatile or non-volatile storage medium comprising digital images respectively depicting a tissue sample of a patient;
an image splitting module being executable by the at least one processor and being configured to split each of the images into a set of image tiles;
a feature extraction module being executable by the at least one processor and being configured to compute, for each of the tiles, a feature vector comprising image features extracted selectively from the said tile;
a Multiple-Instance-Learning (MIL) program being executable by the at least one processor and being configured to use a model for classifying any input image as a member of one out of at least two different classes based on the feature vectors extracted from all tiles of the said input image, wherein the MIL-program is a neural network, and wherein the MIL-program is further configured for:
for each of the tiles, computing a certainty value, the certainty value being indicative of the certainty of the model regarding the contribution of the tile's feature vector on the classification of the image from which the tile was derived, wherein the certainty-value is computed using a dropout technique at test time of the model of the neural network;
for each of the images:
using, by the MIL-program, a certainty-value-based pooling function for aggregating the feature vectors extracted from the image into a global feature vector as a function of the certainty values of the tiles of the image, and computing an aggregated predictive value from the global feature vector; or
computing, by the MIL program, a predictive value from each of the feature vectors of the image and using, by the MIL-program, a certainty-value-based pooling function for aggregating the predictive values of the image into an aggregated predictive value as a function of the certainty values of the tiles of the image; and
classifying, by the MIL-program, each of the images as a member of one out of the at least two different classes based on the aggregated predictive value.
19 . The method of claim 11 , wherein the non-biomarker specific stain is a hematoxylin stain or an H&E stain.
20 . The method of claim 15 , wherein the particular response is a pathologic complete response, and wherein the particular genetic attribute is a particular gene signature.