Systems and methods for deep orthogonal fusion for multimodal prognostic biomarker discovery
A system and method are provided for identifying a multimodal biomarker of a prognostic prediction, using a deep learning framework trained to analyze different modality data, including radiomic image data, pathology image data, and molecular image data to obtain unimodal embedding predictions from those modality data and generate multimodal embedding predictions, through application of a loss minimization and attention-based fusion processes.
1. A computer-implemented method for identifying a multimodal biomarker of a prognostic prediction for a tumor sample, the method comprising:
a) obtaining, using one or more processors, a radiomic dataset for a tumor sample, the radiomic dataset comprising characteristics of a lesion and/or associated tissue and subsequent analysis and classification of the characteristic corresponding to the tumor sample;
b) obtaining, using the one or more processors, a pathology dataset for the tumor sample, the pathology dataset comprising characteristics of cells, cell types, cell shapes, and/or cell areas corresponding to the tumor sample;
c) obtaining, using the one or more processors, a molecular dataset for the tumor sample, the molecular dataset comprising data derived from sequencing data and/or subsequent analysis and classification of the sequencing data corresponding to the tumor sample;
d) providing, using the more or more processors, the radiomic dataset, the pathology dataset, and the molecular modality dataset to a trained deep learning framework and
e) generating, using the trained deep learning framework, a radiomic embedding prediction, a pathology embedding prediction, and a molecular embedding prediction; and
f) applying, using the trained deep learning framework, the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction to a loss minimization to reduce unimodal embeddings for each prior to an embedding fusion to generate a multimodal embedding prediction as the multimodal biomarker corresponding to the prognostic prediction for the tumor sample,
wherein the trained deep learning framework comprises a unimodal embeddings layer for generating the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction and a fully-connected output layer for generating the multimodal embedding prediction, and
wherein the trained deep learning framework is configured to apply, subsequent to the loss minimization at the unimodal embeddings layer, a second loss function at the fully-connected output layer to generate the multimodal embedding prediction as the multimodal biomarker.
2. The method of claim 1 , wherein the radiomic dataset is selected from the group consisting of a magnetic resonance imaging (MRI) image dataset, a computed tomography (CT) image dataset, a fluorescence image dataset, and an x-ray image dataset.
3. The method of claim 1 , wherein the pathology dataset is selected from the group consisting of a hematoxylin and eosin (H&E) stained slide image dataset, an immunohistochemistry (IHC) stained slide image dataset, and a fluorescence in situ hybridization (FISH) image dataset.
4. The method of claim 1 , wherein the molecular dataset is selected from the group consisting of gene sequencing data, RNA data, DNA data, methylation data, and proteomic data.
5. The method of claim 1 , wherein generating, using the trained deep learning framework, the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction comprises:
i) feeding, within the trained deep learning framework, the radiomic dataset to a radiomic neural network trained to generate the radiomic embedding prediction;
ii) feeding, within the trained deep learning framework, the pathology dataset to a pathology neural network trained to generate the pathology embedding prediction; and
iii) feeding, within the trained deep learning framework, the molecular dataset to a molecular neural network trained to generate the molecular embedding prediction.
6. The method of claim 5 , wherein the radiomic neural network is a convolutional neural network.
7. The method of claim 6 , wherein the convolutional neural network has been trained using multiparametric MRI training images and labeled image features.
8. The method of claim 7 , wherein the convolutional neural network comprises a T1 trained convolutional neural network branch, a T2 trained convolutional neural network branch, and a labeled image features branch.
9. The method of claim 5 , wherein the pathology neural network is a convolutional neural network.
10. The method of claim 5 , wherein the molecular neural network is a self-normalizing neural network.
11. The method of claim 1 , wherein the trained deep learning framework is configured to apply, as the loss minimization, a unimodal loss minimization for the unimodal embeddings layer.
12. The method of claim 1 , wherein the trained deep learning framework is configured to apply, as the loss minimization, a multimodal orthogonalization loss across each of the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction.
13. The method of claim 1 , further comprising:
e) performing, using the trained deep learning framework, a multimodal fusion on the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction;
f) generating a multidimensional fusion matrix containing a plurality of multidimensional embeddings, containing at least one or more bi-modal embeddings or one or more tri-modal embeddings; and
g) determining the multimodal embedding prediction from a comparison of the plurality of multidimensional embeddings.
14. The method of claim 1 , further comprising
h) generating the multimodal embedding prediction is a prediction of overall survival rate corresponding to the tumor sample.
15. The method of claim 1 , further comprising
i) generating the multimodal embedding prediction by:
A) generating a plurality of multimodal embeddings each having a prediction score; and
B) identifying a maximum prediction score as the multimodal embedding prediction.
16. The method of claim 1 , further comprising,
j) using the loss minimization,
k) applying a multimodal orthogonalization across the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction.
17. The method of claim 16 , further comprising
l) applying an attention weighting to the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction and, in response,
m) performing the embedding fusion to generate the multimodal embedding prediction.
18. The method of claim 1 , further comprising:
n) receiving additional features from the radiomic dataset, the pathology dataset, and/or the molecular dataset, the additional features not being used by the deep learning framework in generating the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction; and
o) using, in the trained deep learning framework, the additional features to generate the multimodal embedding prediction.
19. A system for identifying a multimodal biomarker of a prognostic prediction for a tumor sample, the system comprising:
I) one or more processors; and
II) a trained deep learning framework application including computing instructions configured to be executed by the one or more processors to;
A) receive a radiomic image modality dataset for a tumor sample, a pathology image modality dataset for the tumor sample, and a molecular modality dataset for the tumor sample, wherein the radiomic dataset comprises characteristics of a lesion and/or associated tissue and subsequent analysis and classification of the characteristic corresponding to the tumor sample, the pathology dataset comprises characteristics of cells, cell types, cell shapes, and/or cell areas corresponding to the tumor sample, and the molecular dataset comprises data derived from sequencing data and/or subsequent analysis and classification of the sequencing data corresponding to the tumor sample;
B) generate a radiomic embedding prediction, a pathology embedding prediction, and a molecular embedding prediction; and
C) from the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction, applying a loss minimization to reduce unimodal embeddings for each prior to an embedding fusion, generate a multimodal embedding prediction as the multimodal biomarker corresponding to the prognostic prediction for the tumor sample,
wherein the trained deep learning framework comprises a unimodal embeddings layer for generating the radiomic embedding prediction, the pathology embedding prediction, and the molecular embedding prediction and a fully-connected output layer for generating the multimodal embedding prediction, and
wherein the trained deep learning framework is configured to apply, subsequent to the loss minimization at the unimodal embeddings layer, a second loss function at the fully-connected output layer to generate the multimodal embedding prediction as the multimodal biomarker.