Enhanced multi-protocol analysis via intelligent supervised embedding (empravise) for multimodal data fusion
View Patent ↗The present invention provides a system and method for analysis of multimodal imaging and non-imaging biomedical data, using a multi-parametric data representation and integration framework. The present invention makes use of (1) dimensionality reduction to account for differing dimensionalities and scale in multimodal biomedical data, and (2) a supervised ensemble of embeddings to accurately capture maximum available class information from the data.
1. A method for analyzing multimodal data derived from a biological sample, comprising:
(a) inputting, via a computing device, the multimodal data comprising a plurality of data;
(b) generating, via the computing device, a plurality of quantitative descriptors in the multimodal data;
(c) constructing, via the computing device, a higher dimensional multi-parametric feature space using the plurality of quantitative descriptors;
(d) reducing, via the computing device, the higher dimensional multi-parametric feature space to a lower dimensional ensemble embedding by an ensemble embedding method comprising:
(i) generating, via the computing device, a plurality of base embeddings derived from the higher dimensional multi-parametric feature space, wherein the plurality of base embeddings comprise weak embeddings and strong embeddings;
(ii) calculating, via the computing device, an embedding strength for each base embedding;
(iii) selecting, via the computing device, a plurality of the strong embeddings, wherein the embedding strength of each strong embedding is greater than a pre-specified threshold value;
(iv) constructing, via the computing device, the lower dimensional ensemble embedding by:
(1) quantifying, via the computing device, a plurality of pair-wise relationships for all quantitative descriptors in each strong embedding in step (iii) via a Euclidean pair-wise distance method;
(2) normalizing, via the computing device, the plurality of pair-wise relationships to yield a plurality of normalized relationships;
(3) propagating, via the computing device, the plurality of normalized relationships by a projection method; and
(4) obtaining, via the computing device, the lower dimensional ensemble embedding;
(v) optimizing, via the computing device, the generating a plurality of independent embeddings step (i), the calculating an embedding strength step (ii), the selecting a plurality of strong embeddings step (iii), and the constructing the lower dimensional ensemble embedding step (iv); and
(e) obtaining, via the computing device, a classification for the biological sample based on the ensemble embedding; and
(f) outputting, via the computing device, the classification for the biological sample.
2. The method according to claim 1 , wherein the multimodal data is an image.
3. The method according to claim 2 , wherein the image is selected from the group consisting of a magnetic resonance imaging (MRI) image, a positron emission tomography (PET) image, a single-photon emission computed tomography (SPECT) image, an ultrasound image, an x-ray image, a computed tomography (CT) image, a micro-array image, and a gene expression image.
4. The method according to claim 1 , wherein the quantitative descriptor is a pixel.
5. The method according to claim 1 , wherein the plurality of base embeddings is generated using space perturbation method comprising:
(1) generating, via the computing device, a plurality of boostrapped feature sub-sets from each quantitative descriptor of the plurality of quantitative descriptors contained within the higher dimensional multi-parametric feature space; and
(2) embedding, via the computing device, a feature space associated with each bootstrapped feature subset of the plurality of boostrapped feature sub-sets into the higher dimensional multi-parametric feature space to create the plurality of independent embeddings.
6. The method according to claim 1 , wherein the plurality of base embeddings is generated by implementing Algorithm CreateEmbed, as given below:
Input: °F(c) R n for all objects c ε C, η
Output: X (c) ε R m η , m ε {1, . . . , M}
Data Structures: Feature subsets η n m , total number of subsets M,
number of features in each subset V, DR method Φ
begin
0.
for m = 1 to M do
1.
Select V < N features from R n , forming subset ηm;
2.
Calculate X m (c) ε R m η , m for all c ε C using n m and
method Φ;
3.
end for
End.
7. The method according to claim 1 , wherein the plurality of base embeddings is generated by implementing Algorithm (CreateWeakEmbed) as follows:
Algorithm CreateWeakEmbed
Input: °F(c) ε F(c) R n for all objects RN for all objects c ε C, η
Output: X m (c) ε R m η , m ε{1, . . . , M}
Data Structures: Feature subsets F m ., total number of subsets M, number
of features in each subset V begin
0.
for m = 1 to M do
1.
Select V < N features from R N , forming subset F m ;
2.
Calculate X m (c) ε R m η , for all c ε C via GE of the feature space
associated with each F m ;
3.
endfor
End.
8. The method according to claim 1 , wherein the plurality of strong embeddings are selected by implementing Algorithm SelErnbed, as follows:
Algorithm SelEmbed
Input: X m (c) ε R m η , for all objects c ε {1, . . . , M}
Output: X m (c) ε R m η , k ε{1, . . . , K}
Data Structures: A list Q, embedding strength function ψ ES , embedding
strength threshold θ
begin
0.
for m = 1 to M do
1.
Calculate ψ ES (R m η ,);
2.
If ψ ES (R m η ,) > θ;
3.
Put m in Q;
4.
endif
5.
endfor
6.
For each element k of Q, store X k (c) ε R m η , for all objects c ε C;
End.
9. The method according to claim 1 , wherein the lower dimensional embedding is constructed by implementing Algorithm CalcEnsembleEmbed:
Algorithm CalcEnsembleEmbed
Input: X k (c) ε R m η , for all objects c ε C; k ε {1, . . . , k}
Output: X (c) = R N
Data Structures: Confusion matrix W, ML estimator, Ω, projection method γ
begin
0.
for k k = 1 to k do
1.
Calculate W k (i, j) ∥ X k (d) ∥ 2 for all objects c, d ε C with
indices i, j
2.
endfor
3.
Apply normalization to all W k , k ε {1, . . . , k}
4.
Obtain ω (i, j) = Ω [W k (i, j)] ∇ c, d ε C;
5.
Apply projection method γ to w to obtain final ensemble embedding R N ;
End.
10. The method according to claim 1 , wherein the biological sample is derived from a diseased tissue.
11. The method according to claim 10 , wherein the diseased tissue is a cancer tissue.
12. The method according to claim 11 , wherein the cancer is selected from the group consisting of a brain cancer, a prostate cancer, a breast cancer, a pancreatic cancer, a colon cancer, a leukemia, a lymphoma, a myeloma, a ovarian cancer, a skin cancer, a lung cancer, and a renal-cell cancer.
13. A method for analyzing multimodal data derived from a biological sample by a multi-protocol analysis using an intelligent supervised embedding (EMPrAvISE) scheme, comprising:
(a) inputting, via a computing device, the multimodal data, wherein the multimodal data comprises a plurality of data;
(b) generating, via the computing device, a plurality of voxels in the multimodal data;
(c) assigning, via the computing device, an intensity value to each voxel in the multimodal data;
(d) registering, via the computing device, the plurality of data to create a plurality of registered data, such that the plurality of registered data have the same spatial frame as a reference datum, and computing a ground truth extent of disease for each voxel;
(e) extracting from the plurality of registered data, via the computing device, a plurality of features and computing a plurality of quantitative descriptors corresponding to the plurality of features;
(f) fusing, via the computing device, the plurality of quantitative descriptors to create a plurality of ensemble embeddings by:
(i) constructing, via the computing device, a higher dimensional multi-parametric feature space using the plurality of quantitative descriptors;
(ii) computing, via the computing device, a plurality of pair-wise relationships for all quantitative descriptors of the plurality of quantitative descriptors in the higher dimensional multi-parametric feature space via a Euclidean pairwise distance method and storing the plurality of pair-wise relationships in a confusion matrix;
(iii) generating, via the computing device, a plurality of independent embeddings derived from the higher dimensional multi-parametric feature space, wherein the plurality of independent embeddings comprise weak embeddings and strong embeddings;
(iv) computing, via the computing device, an embedding strength for each independent embedding;
(v) selecting, via the computing device, a plurality of strong embeddings from the plurality of independent embeddings, wherein the embedding strength of each strong embedding is greater than a pre-specified threshold value and wherein a pair-wise relationship between two quantitative descriptors in the higher dimensional multi-parametric feature space is preserved in each strong embedding;
(vi) computing, via the computing device, a maximum likelihood estimate (MLE) of a probability distribution of each pair-wise relationship of the plurality of pairwise relationships in step (v) over the plurality of independent embeddings in the higher dimensional multi-parametric feature space to create a plurality of maximum likelihood estimates (MLEs) corresponding to the plurality of pairwise relationships, and storing the plurality of maximum likelihood estimates (MLEs) in an ensemble matrix; and
(vii) computing, via the computing device, the plurality of ensemble embeddings by a multi-dimensional scaling method, wherein each pair-wise relationship in each ensemble embedding equals the corresponding maximum likelihood estimate (MLE) of the pair-wise relationship in the ensemble matrix;
(g) classifying, via the computing device, the biological sample into a disease class; and
(h) outputting, via the computing device, the disease class.
14. The method according to claim 13 , wherein the multimodal data is an image.
15. The method according to claim 14 , wherein the image is selected from the group consisting of a magnetic resonance imaging (MRI) image, a positron emission tomography (PET) image, a single-photon emission computed tomography (SPECT) image, an ultrasound image, an x-ray image, a computed tomography (CT) image, a micro-array image, and a gene expression image.
16. The method according to claim 15 , wherein the disease is a cancer.
17. The method according to claim 16 , wherein the cancer is selected from a breast cancer, a prostate cancer, a brain cancer, a pancreatic cancer, a colon cancer, a leukemia, a lymphoma, a myeloma, a ovarian cancer, a skin cancer, a lung cancer, and a renal-cell cancer.