Predicting response to immunotherapy using computer extracted features relating to spatial arrangement of tumor infiltrating lymphocytes in non-small cell lung cancer
Embodiments include controlling a processor to perform operations, the operations comprising: accessing a digitized image of a region of tissue demonstrating non-small cell lung cancer (NSCLC), detecting a member of a plurality of cellular nuclei represented in the image; classifying the member of the plurality of cellular nuclei as a tumor infiltrating lymphocyte (TIL) nucleus or non-TIL nucleus; extracting spatial TIL features from the plurality of cellular nuclei, including a first subset of features based on the spatial arrangement of TIL nuclei, and a second, different subset of features based on the spatial relationship between TIL nuclei and non-TIL nuclei; generating a set of graph interplay features based on the set of spatial TIL features; providing the set of graph interplay features to a machine learning classifier; receiving, from the machine learning classifier, a probability that the region of tissue will respond to immunotherapy, based, at least in part, on the set of graph interplay features; classifying the region of tissue as likely to respond to immunotherapy or unlikely to respond to immunotherapy based, at least in part, on the probability; and displaying the classification.
1 . A non-transitory computer-readable storage device storing computer-executable instructions that when executed control a processor to perform operations, the operations comprising:
accessing a digitized image of a region of tissue demonstrating early-stage non-small cell lung cancer (NSCLC), where the region of tissue includes a plurality of cellular nuclei;
detecting a member of the plurality of cellular nuclei represented in the digitized image;
classifying the member of the plurality of cellular nuclei as a tumor infiltrating lymphocyte (TIL) nucleus or a non-TIL nucleus;
extracting a set of spatial TIL features from the plurality of cellular nuclei, where the set of spatial TIL features includes a first subset of features based on the a spatial arrangement of TIL nuclei represented in the digitized image, and a second, different subset of features based on the spatial relationship between TIL nuclei and non-TIL nuclei represented in the digitized image;
generating a set of graph interplay features based on the set of spatial TIL features;
providing the set of graph interplay features to a machine learning classifier pre-trained to determine a probability that a region of tissue will respond to immunotherapy based, at least in part, on the set of gragh interplay features;
receiving, from the machine learning classifier, a the probability that the region of tissue will respond to immunotherapy, where the machine learning classifier computes the probability based, at least in part, on the set of spatial TIL features;
classifying the region of tissue as likely to respond to immunotherapy or unlikely to respond to immunotherapy based, at least in part, on the probability using the machine learning classifier; and
displaying the classification generating and providing a personalized treatment plan for a patient having the region of tissue, the personalized treatment plan being based, at least in part, on the classification.
2 . The non-transitory computer-readable storage device of claim 1 , the operations further comprising:
generating the first subset of features, where generating the first subset of features includes:
generating a TIL cluster graph, where a node of the TIL cluster graph is a centroid of a TIL-nucleus, where the probability that a first centroid of a first TIL-nucleus will be connected with a second, different centroid of a second, different TIL nucleus is based on a weighted Euclidean norm of the a Euclidean distance between the first centroid and the second, different centroid; and
computing a first set of spatial TIL features based on the TIL cluster graph.
3 . The non-transitory computer-readable storage device of claim 2 , where the first set of spatial TIL features includes:
the mean of the a number of TILs in the TIL cluster graph, the mean of the a ratio of the an area of the TIL cluster graph and the an area of the region of tissue represented in the image, and the mean of the a ratio of the number of TILs within the TIL cluster graph and the area of the TIL cluster graph;
the mode of the number of TILs in the TIL cluster graph, the mode of the a ratio of the area of the TIL cluster graph and the area of the region of tissue represented in the image, and the mode of the a ratio of the number of TILs within the TIL cluster graph and the area of the TIL cluster graph; and
the median of the number of TILs in the TIL cluster graph, the median of the a ratio of the area of the TIL cluster graph and the area of the region of tissue represented in the image, and the median of the a ratio of the number of TILs within the TIL cluster graph and the area of the TIL cluster graph.
4 . The non-transitory computer-readable storage device of claim 2 , the operations further comprising:
generating the second subset of features, where generating the second subset of features includes:
generating a non-TIL cluster graph where a node of the non-TIL cluster graph is a centroid of a non-TIL-nucleus, where the probability that a first centroid of a first non-TIL nucleus will be connected with a second, different centroid of a second, different non-TIL nucleus is based on a weighted Euclidean norm of the a Euclidean distance between the first non-TIL centroid of the first non-TIL nucleus and the second, different non-TIL centroid of the second, different non-TIL nucleus; and
computing a second set of spatial TIL features based on the non-TIL cluster graph and the TIL cluster graph, where the second set of spatial TIL features describes a statistical relationship between the non-TIL cluster graph and the TIL cluster graph.
5 . The non-transitory computer-readable storage device of claim 4 , where the second set of spatial TIL features includes:
a ratio between the a density of a non-TIL cluster graph and the a density of the TIL cluster graph nearest the non-TIL cluster graph;
the a value of an intersecting area of a the non-TIL cluster graph and a the TIL cluster graph; and
a value indicating if the a nearest neighbor cluster graph of a the TIL cluster graph is a non-TIL cluster graph or a TIL cluster graph.
6 . The non-transitory computer-readable storage device of claim 1 , where the machine learning classifier is a random forest classifier.
7 . The non-transitory computer-readable storage device of claim 6 , where the set of graph interplay features includes:
a weighted efficiency between a first two graphs feature;
a similarity between a second two graphs feature;
a nodes closeness ratio;
a mean number of lymphocyte clusters in a cancer cluster feature; and
a difference of characteristic path length between a third two graphs feature.
8 . The non-transitory computer-readable storage device of claim 1 , where the machine learning classifier is a quadratic discriminant analysis (QDA) classifier.
9 . The non-transitory computer-readable storage device of claim 8 , where the set of graph interplay features includes:
an area of TIL clusters feature;
a spatial proximity of TILs feature;
a spatial proximity of TILs to non-TIL feature; and
a density of clusters feature.
10 . The non-transitory computer-readable storage device of claim 1 , where detecting a the member of the plurality of cellular nuclei represented in the digitized image includes detecting the member of the plurality of cellular nuclei represented in the digitized image using a watershed segmentation technique, where the watershed segmentation technique includes applying, at a plurality of scales, a fast radial symmetry transform and regional minima to the digitized image.
11 . The non-transitory computer-readable storage device of claim 1 , where classifying the member of the plurality of cellular nuclei as a the TIL nucleus or the non-TIL nucleus comprises:
extracting a set of image features from the member of the plurality of cellular nuclei, where the set of image features includes a texture feature, a shape feature, and a color feature;
providing the set of image features to a second, different, machine learning classifier; and
receiving, from the second machine learning classifier, a classification of the member of the plurality of cellular nuclei as a the TIL nucleus or the non-TIL nucleus, where the second machine learning classifier classifies the member of the plurality of cellular nuclei based on the set of image features.
12 . The non-transitory computer-readable storage device of claim 1 , where the digitized image is a 1500 pixel by 1500 pixel digitized hematoxylin and eosin (H&E) stained image scanned at 20× magnification.
13 . The non-transitory computer-readable storage device of claim 1 , the operations further comprising displaying the classification and at least one of the digitized image, the probability, the first subset of features, the second subset of features, or the set of graph interplay features.
14 . An apparatus for predicting response to immunotherapy in early-stage non-small cell lung cancer (NSCLC), comprising:
a processor;
a memory configured to store a digitized image of a hematoxylin and eosin (H&E) stained image of a region of tissue demonstrating early stage NSCLC;
an input/output (I/O) interface;
a set of circuits; and
an interface that connects the processor, the memory, the I/O interface, and the set of circuits, the set of circuits comprising:
an image acquisition circuit configured to access a diagnostic image of a region of tissue demonstrating early-stage NSCLC, where the region of tissue includes a plurality of cellular nuclei;
a nuclei detecting and segmentation circuit configured to:
detect a member of the plurality of cellular nuclei represented in the diagnostic image; and
classify the member of the plurality of cellular nuclei as a tumor infiltrating lymphocyte (TIL) nucleus or non-TIL nucleus;
a spatial TIL feature circuit configured to:
extract a set of spatial TIL features from the plurality of cellular nuclei represented in the diagnostic image, where the set of spatial TIL features includes a first subset of features based on the a spatial arrangement of TIL nuclei represented in the diagnostic image, and
extract a second subset of features based on the a spatial relationship between TIL nuclei and non-TIL nuclei represented in the diagnostic image;
a graph interplay circuit configured to compute a set of graph interplay features based on the set of spatial TIL features;
an immunotherapy response classification circuit configured to:
compute a probability that the region of tissue will respond to immunotherapy, based, at least in part, on the set of graph interplay features using a machine learning classifier pre-trained to compute the probability, based, at least in part, on the set of gragh interplay features; and
generate a classification of the region of tissue as likely to respond to immunotherapy, or unlikely to respond to immunotherapy, based, at least in part, on the probability; and
a display circuit configured to display the classification and at least one of the probability, the diagnostic image, the set of spatial TIL features, or the set of graph interplay features.
15 . The apparatus of claim 14 , where the nuclei detecting and segmentation circuit is configured to detect the member of the plurality of cellular nuclei represented in the diagnostic image using a watershed segmentation technique, where the watershed segmentation technique includes applying, at a plurality of scales, a fast radial symmetry transform and regional minima to the diagnostic image.
16 . The apparatus of claim 14 , where the spatial TIL feature circuit is further configured to generate the first subset of features by:
generating a TIL cluster graph, where a node of the TIL cluster graph is a centroid of a TIL-nucleus, where the probability that a first centroid of a first TIL-nucleus will be connected with a second, different centroid of a second, different TIL-nucleus is based on a weighted Euclidean norm of the a Euclidean distance between the first centroid of the first TIL-nucleus and the second centroid of the second, different TIL-nucleus; and
computing a first set of spatial TIL features based on the TIL cluster graph.
17 . The apparatus of claim 16 , where the spatial TIL feature circuit is further configured to generate the second subset of features by:
generating a non-TIL cluster graph, where a node of the non-TIL cluster graph is a centroid of a non-TIL nucleus, where the probability that a first centroid of a first, non-TIL nucleus will be connected with a second, different centroid of a second, different non-TIL nucleus is based on a weighted Euclidean norm of the a Euclidean distance between the first centroid of the first non-TIL nucleus and the second centroid of the second, different non-TIL nucleus; and
computing a second set of spatial TIL features based on the non-TIL cluster graph and the TIL cluster graph, where the second set of spatial TIL features describes a statistical relationship between the non-TIL cluster graph and the TIL cluster graph.
18 . The apparatus of claim 14 , where the immunotherapy response classification circuit is configured to compute the probability that the region of tissue will respond to immunotherapy using a random forest classifier trained on a set of training images, where a member of the set of training images is acquired using different imaging parameters than the diagnostic image.
19 . The apparatus of claim 14 , where the set of graph interplay features includes:
a weighted efficiency between two graphs feature;
a similarity between two graphs feature;
a nodes closeness ratio;
a mean number of lymphocyte clusters in a cancer cluster feature; and
a difference of characteristic path length between two graphs feature.
20 . A non-transitory computer readable storage device storing computer-executable instructions that when executed by a computer control the computer to perform a method for predicting response to immunotherapy in early-stage non-small cell lung cancer (NSCLC), the method comprising:
accessing a scanned image of a hematoxylin and eosin (H&E) stained slide of a region of tissue demonstrating early stage NSCLC, where the region of tissue includes a plurality of cellular nuclei, where the scanned image is acquired using a first set of imaging parameters;
detecting, using a watershed segmentation technique, a member of the plurality of cellular nuclei represented in the scanned image;
extracting a set of image features from the member of the plurality of cellular nuclei;
classifying the member of the plurality of cellular nuclei as a tumor infiltrating lymphocyte (TIL) nucleus or non-TIL nucleus based, at least in part, on the set of image features;
extracting a set of spatial TIL features from the plurality of cellular nuclei, where the set of spatial TIL features includes a first subset of features based on the a spatial arrangement of TIL nuclei represented in the scanned image, and a second subset of features based on the a spatial relationship between TIL nuclei and non-TIL nuclei represented in the scanned image;
generating a set of graph interplay features based on the set of spatial TIL features, where the set of graph interplay features includes a weighted efficiency between two graphs feature, a similarity between two graphs feature, a nodes closeness ratio, a mean number of lymphocyte clusters in a cancer cluster feature, and a difference of characteristic path length between two graphs feature;
computing, using a random forest classifier, a probability that the region of tissue will respond to immunotherapy, where the random forest classifier computes the probability based, at least in part, on the set of graph interplay features, where the random forest classifier is trained on pre-trained to compute the probability using a training data set of scanned images of H&E stained slides of a region of tissue demonstrating early stage NSCLC, where at least one member of the training data set is acquired using a second, different set of imaging parameters;
generating a classification of the region of tissue as likely to respond to immunotherapy or unlikely to respond to immunotherapy based, at least in part, on the probability; and
displaying the classification and at least one of the probability, the scanned image, or the set of graph interplay features; and
generating and providing a personalized treatment plan for a patient having the region of tissue, the personalized treatment plan being based, at least in part, on the classification.
21. A method, comprising:
generating a first mathematical description of a spatial arrangement of lymphocytes based on an image dataset of a region of tissue demonstrating a cancer;
generating a second mathematical description of a spatial arrangement of nonlymphocyte cells based on the image dataset;
generating a set of interplay features describing a spatial relationship between lymphocytes and non-lymphocyte cells based on the first mathematical description and the second mathematical description;
providing the set of interplay features to a classifier function pre-trained to generate a probability that the region of tissue is likely to respond to immunotherapy based, at least in part, on the set of interplay features; and
outputting at least the probability.
22. The method of claim 21 , wherein the lymphocytes are tumor-infiltrating lymphocytes (TILs) and the non-lymphocyte cells are non-TILs.
23. The method of claim 22 , wherein:
the first mathematical description comprises one or more graphs of TIL nuclei in the image dataset; and
the second mathematical description comprises one or more graphs of non-TIL nuclei in the image dataset.
24. The methond of claim 23 , wherein:
nodes of each of the one or more graphs of TIL nuclei comprise centroids of the TIL nuclei;
nodes of each of the one or more graphs of non-TIL nuclei comprise centroids of the non-TIL nuclei;
a probability that any two nodes of one of the one or more graphs of TIL nuclei will be connected to one another depends on a Euclidean distance between the two nodes; and
a probability that any two nodes of one of the one or more graphs of non-TIL nuclei will be connected to one another depends on a Euclidean distance between the two nodes.
25. The method of claim 24 , further comprising:
performing segmentation on the image dataset to identify the TILs and the non-TIL cells in the image dataset.
26. The method of claim 25 , wherein performing the segmentation comprises identifying the TIL nuclei and the non-TIL nuclei.
27. The method of claim 23 , wherein the set of interplay features comprises one or more of a weighted efficiency between two graphs, a similarity between two graphs, a nodes closeness ratio, a mean number of lymphocyte clusters in a cancer cluster, and a difference of characteristic path length between two graphs.
28. The method of claim 21 , wherein the image dataset is a digitized hematoxylin and eosin (H&E) stained pathology image.
29. A non-transitory computer-readable storage device storing computer-executable instructions that when executed control a processor to perform the method of claim 21 .