MULTI-MODAL APPROACH TO PREDICTING IMMUNE INFILTRATION BASED ON INTEGRATED RNA EXPRESSION AND IMAGING FEATURES
Multi-modal approaches to predict tumor immune infiltration are based on integrating gene expression data and imaging features in a neural network-based framework. This framework is configured to estimate percent composition, and thus immune infiltration score, of a patient tumor biopsy sample. Multi-modal approaches may also be used to predict cell composition beyond immune cells via integrated multi-layer neural network frameworks.
1 . A computing device configured to generate an immune infiltration prediction score, the computing device comprising one or more processors configured to:
obtain gene expression data from one or more gene expression datasets with the gene expression data corresponding to one or more tissue samples;
obtain a set of stained histopathology images from one or more image sources and corresponding to the one or more tissue samples;
determine imaging features from the set of stained histopathology images, the imaging features comprising texture and/or intensity features;
in a neural network framework, transform the gene expression data using a gene expression neural network layer(s) and transform the imaging features using an imaging feature neural network layer(s);
in the neural network framework, integrate an output of the gene expression neural network layer(s) and the imaging feature neural network layer(s) to produce an integrated neural network output; and
apply a prediction function to the integrated neural network output and output an immune infiltration score for the one or more tissue samples.
2 .- 3 . (canceled)
4 . The computing device of claim 1 , wherein the gene expression data is RNA sequencing data.
5 . The computing device of claim 1 , wherein the neural network framework comprises two neural network layers.
6 . The computing device of claim 1 , wherein the imaging features comprise mean, standard deviation, skewness, and/or sum of image gray level, image red, green, blue layers, stain layers, optical density, hue, and/or saturation.
7 . The computing device of claim 1 , wherein the imaging features comprise Zernike moments, threshold adjacency analysis values, local binary patterns, gray scale co-occurrence matrix, and/or difference of Gaussian statistical measures.
8 . The computing device of claim 1 , wherein the prediction function is Softmax function.
9 . The computing device of claim 1 , wherein immune infiltration score comprises a predicted percentage of natural killer (NK) cells, (MAC) macrophage cells, CD4 T cells, CD8 T cells, and B cells, regulatory T cells, Dendritic cells, monocytes, Mast cells, Eosinophils, and Neutrophils.
10 . The computing device of claim 1 , wherein immune infiltration score comprises a predicted percentage of others cells, including stromal cells, vasculature cells, fat cells, tumor cells, stem cells, neural cells, progenitor cells, innate lymphoid cells, microglial cells, leukocytes, naïve B cells, memory B cells, Plasma cells, CD8 T cells, naïve CD4 T cells, memory CD4 T cells, follicular helper T cells, regulatory T cells, gamma delta T cells, Th17 T cells, unstimulated NK cells, stimulated NK cells, Macrophages MO, Macrophages M1, Macrophages M2, unstimulated Dendritic cells, stimulated Dendritic cells, unstimulated Mast cells, stimulated Mast cells.
11 . The computing device of claim 1 , wherein the contextual data is a total immune fraction or total tumor fraction.
12 . A computer-implemented method to generate an immune infiltration prediction score, the method comprising:
obtaining a gene expression data from one or more gene expression datasets with the gene expression data corresponding to one or more tissue samples;
obtaining a set of stained histopathology images from one or more image sources and corresponding to the one or more tissue samples;
determining imaging features from the set of stained histopathology images, the imaging features comprising texture and/or intensity features;
in a neural network framework, transforming the gene expression data using a gene expression neural network layer(s) and transforming the imaging features using an imaging feature neural network layer(s);
in the neural network framework, integrating an output of the gene expression neural network layer(s) and the imaging feature neural network layer(s) to produce an integrated neural network output; and
applying a prediction function to the integrated neural network output and outputting an immune infiltration score for the one or more tissue samples.
13 .- 14 . (canceled)
15 . The computer-implemented method of claim 12 , further comprising obtaining the gene expression data from an RNA sequencing data source communicatively coupled to a communication network, the gene expression data comprising RNA sequencing data.
16 . The computer-implemented method of claim 15 , wherein the gene expression data is raw gene expression data.
17 . The computer-implemented method of claim 15 , wherein the gene expression data is RNA sequencing data for selected genes.
18 . The computer-implemented method of claim 12 , wherein the gene expression data is RNA sequencing data, the method further comprising performing a gene selection on the RNA sequencing data.
19 . The computer-implemented method of claim 12 , further comprising: tilting each of the set of stained histopathology images to generate a plurality of patches; and determining the imaging features from the plurality of patches.