Method for predicting microsatellite instability (MSI) from whole slide image based on self-attention mechanism
A method for predicting a microsatellite instability (MSI) from a whole slide image based on a self-attention mechanism, in which an image of a tissue region obtained through cellular staining on a tissue sample of a patient serves as a target whole slide image to be predicted. The tissue region is segmented into tissue patches, and stain normalization is performed with staintools. N tumor patches are screened out from the tissue patches, and the N tumor patches are inputted to a preset feature extraction model. Each of the tumor patches are encoded as an M-dimensional feature vector, thereby obtaining a N×M feature matrix; and the N×M feature matrix is inputted to a pre-trained prediction model, such that the prediction model outputs an MSI score of the target whole slide image. The present disclosure predicts the MSI based on deep learning and the cellular staining.
1 . A method for predicting a microsatellite instability (MSI) from a whole slide image based on a self-attention mechanism, comprising the following steps:
S 1 : acquiring a target whole slide image to be predicted, the target whole slide image being an image of a tissue region obtained through cellular staining on a tissue sample of a patient;
S 2 : segmenting the tissue region in the target whole slide image into a plurality of tissue patches having a preset resolution, and performing stain normalization with staintools;
S 3 : determining tumor probabilities of the tissue patches, and screening out N tissue patches with probabilities satisfying a preset condition to serve as tumor patches;
S 4 : inputting the N tumor patches to a preset feature extraction model, and encoding each of the tumor patches as an M-dimensional feature vector, thereby obtaining a N×M feature matrix; and
S 5 : inputting the N×M feature matrix to a pre-trained prediction model, such that the prediction model outputs an MSI score of the target whole slide image, wherein
the prediction model is a model obtained by training a self-attention model based on a first sample set, and the first sample set comprises a feature matrix of a pathological sample image and an MSI score corresponding to the pathological sample image.
2 . The method for predicting an MSI from a whole slide image based on a self-attention mechanism according to claim 1 , wherein a method that the prediction model predicts the MSI score of the target whole slide image in step S 5 comprises:
S 51 : transforming the M-dimensional feature vector corresponding to the tumor patch into a low-dimensional feature vector;
S 52 : performing attention pooling on the low-dimensional feature vector corresponding to the tumor patch to obtain a weight of the tumor patch;
S 53 : seeking a weighted sum for low-dimensional feature vectors corresponding to all of the tumor patches to obtain a low-dimensional feature vector of the whole target whole slide image; and
S 54 : mapping the low-dimensional feature vector of the whole target whole slide image to a classification space, and performing softmax activation to obtain the MSI score.
3 . The method for predicting an MSI from a whole slide image based on a self-attention mechanism according to claim 2 , wherein the M-dimensional feature vector is a 1,024-dimensional vector, and the low-dimensional feature vector is a 512-dimensional vector.
4 . The method for predicting an MSI from a whole slide image based on a self-attention mechanism according to claim 2 , wherein in step S 51 , the M-dimensional feature vector corresponding to the tumor patch is transformed into the low-dimensional feature vector by:
h k =ReLU(W 2 (ReLU (W 1 Z k +b 1 ))+b 2 )
wherein, Z k denotes an M-dimensional feature vector of a kth tumor patch, h k denotes a low-dimensional feature vector of the kth tumor patch, W 1 and W 2 denote a weight parameter of a first fully- connected (FC) layer, and b 1 and b 2 denote a deviation parameter of the first FC layer.
5 . The method for predicting an MSI from a whole slide image based on a self-attention mechanism according to claim 2 , wherein in step S 51 , the weight of the tumor patch is obtained by:
a
k
=
exp
{
W
a
(
tanh
(
Vh
k
+
c
)
⊙
sigm
(
Uh
k
+
d
)
)
}
∑
j
=
1
N
exp
{
W
a
(
tanh
(
Vh
j
+
c
)
⊙
sigm
(
Uh
j
+
d
)
)
}
wherein, a k denotes a weight of a kth tumor patch, V and U denote a weight parameter of a second FC layer, c and d denote a deviation parameter of the second FC layer, and h k denotes a low-dimensional feature vector of the kth tumor patch.
6 . The method for predicting an MSI from a whole slide image based on a self-attention mechanism according to claim 2 , wherein in step S 54 , the MSI score is obtained by:
p=softmax (W cls h slide +b cls )
wherein, p denotes the MSI score, W cls denotes a weight parameter of a third FC layer, b cls denotes a deviation parameter of the third FC layer, and h slide denotes the low-dimensional feature vector of the whole target whole slide image.
7 . The method for predicting an MSI from a whole slide image based on a self-attention mechanism according to claim 1 , wherein the feature extraction model is a convolutional neural networks (CNN) model with a VGGNet, AlexNet, InceptionNet or ResNet architecture.
8 . The method for predicting an MSI from a whole slide image based on a self-attention mechanism according to claim 7 , wherein the feature extraction model uses a ResNet50 model.
9 . The method for predicting an MSI from a whole slide image based on a self-attention mechanism according to claim 1 , wherein the determining tumor probabilities of the tissue patches in step S 3 comprises:
inputting the tissue patches obtained in step S 2 to a pre-trained classification model, such that the classification model outputs the tumor probabilities and normal probabilities of the tissue patches, wherein
the classification model is a model obtained by training CNN based on a second sample set, and the second sample set comprises a plurality of tumor patches and a plurality of normal patches randomly selected from the pathological sample image.
10 . The method for predicting an MSI from a whole slide image based on a self-attention mechanism according to claim 9 , wherein the classification model uses a ResNet18 model.
11 . An electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
Wherein the memory is configured to store a computer program; and
the processor is configured to execute the computer program in the memory to implement steps of the method for predicting a microsatellite instability (MSI) from a whole slide image based on a self-attention mechanism according to claim 1 .
12 . The electronic device according to claim 11 , wherein a method that the prediction model predicts the MSI score of the target whole slide image in step S 5 comprises:
S 51 : transforming the M-dimensional feature vector corresponding to the tumor patch into a low-dimensional feature vector;
S 52 : performing attention pooling on the low-dimensional feature vector corresponding to the tumor patch to obtain a weight of the tumor patch;
S 53 : seeking a weighted sum for low-dimensional feature vectors corresponding to all of the tumor patches to obtain a low-dimensional feature vector of the whole target whole slide image; and
S 54 : mapping the low-dimensional feature vector of the whole target whole slide image to a classification space, and performing softmax activation to obtain the MSI score.
13 . The electronic device according to claim 12 , wherein the M-dimensional feature vector is a 1,024-dimensional vector, and the low-dimensional feature vector is a 512-dimensional vector.
14 . The electronic device according to claim 12 , wherein in step S 51 , the M-dimensional feature vector corresponding to the tumor patch is transformed into the low-dimensional feature vector by:
h k =ReLU(W 2 (ReLU(W 1 Z k +b 1 ))+b 2 )
wherein, Z k denotes an M-dimensional feature vector of a kth tumor patch, h k denotes a low-dimensional feature vector of the kth tumor patch, W 1 and W 2 denote a weight parameter of a first fully-connected (FC) layer, and b 1 and b 2 denote a deviation parameter of the first FC layer.
15 . The electronic device according to claim 12 , wherein in step S 51 , the weight of the tumor patch is obtained by:
wherein, a k denotes a weight of a kth tumor patch, V and U denote a weight parameter of a second FC layer, c and d denote a deviation parameter of the second FC layer, and h k denotes a low-dimensional feature vector of the kth tumor patch.
16 . The electronic device according to claim 12 , wherein in step S 54 , the MSI score is obtained by:
p=softmax(W cls h slide +b cls )
wherein, p denotes the MSI score, W cls denotes a weight parameter of a third FC layer, b cls denotes a deviation parameter of the third FC layer, and hslide denotes the low-dimensional feature vector of the whole target whole slide image.
17 . The electronic device according to claim 11 , wherein the feature extraction model is a convolutional neural networks (CNN) model with a VGGNet, AlexNet, InceptionNet or ResNet architecture.
18 . The electronic device according to claim 17 , wherein the feature extraction model uses a ResNet50 model.
19 . The electronic device according to claim 11 , wherein the determining tumor probabilities of the tissue patches in step S 3 comprises:
inputting the tissue patches obtained in step S 2 to a pre-trained classification model, such that the classification model outputs the tumor probabilities and normal probabilities of the tissue patches, wherein
the classification model is a model obtained by training CNN based on a second sample set, and the second sample set comprises a plurality of tumor patches and a plurality of normal patches randomly selected from the pathological sample image.
20 . A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores a computer program, and the computer program is executed by a processor configured to implement steps of the method for predicting a microsatellite instability (MSI) from a whole slide image based on a self-attention mechanism according to claim 1 .