Systems and methods for processing images of slides to infer biomarkers
Systems and methods are disclosed for receiving a target electronic image corresponding to a target specimen, the target specimen comprising a tissue sample of a patient, applying a machine learning system to the target electronic image to identify a region of interest of the target specimen and determine an expression level of, category of, and/or presence of a biomarker in the region of interest, the biomarker comprising at least one from among an epithelial growth factor receptor (EGFR) biomarker and/or a DNA mismatch repair (MMR) deficiency biomarker, the machine learning system having been generated by processing a plurality of training images to predict whether a region of interest is present in the target electronic image, the training images comprising images of human tissue and/or images that are algorithmically generated, and outputting the determined expression level of, category of, and/or presence of the biomarker in the region of interest.
1. A computer-implemented method for analyzing an image corresponding to a specimen, the method comprising:
receiving a target electronic image corresponding to a target specimen, the target specimen comprising a tissue sample of a patient;
applying a first machine learning algorithm to the target electronic image to identify one or more salient regions of the target electronic image;
applying a second machine learning algorithm focusing on the one or more salient regions of the target electronic image to identify a region of interest of the target specimen and determine a numeric expression level of, category of, and a numeric probability that there is a presence of a biomarker in the region of interest of the target specimen, the biomarker comprising at least one from among an epithelial growth factor receptor (EGFR) biomarker and/or a DNA mismatch repair (MMR) deficiency biomarker, the second machine learning algorithm having been generated by processing a plurality of training images to predict whether a region of interest is present in the target electronic image, the training images comprising images of human tissue and/or images that are algorithmically generated; wherein determining the numeric expression level of the biomarker includes:
breaking the one or more salient regions of target electronic image into a plurality of tiles;
determining, by the second machine learning algorithm, the numeric expression level corresponding to a prediction for each tile to determine a plurality of tile predictions; and
aggregating the plurality of tile predictions into at least one part level numeric expression level prediction for sub-regions of the one or more salient regions, wherein the at least one part level numeric expression level prediction and prediction for each tile are fed into the second machine learning algorithm to determine the numeric expression level prediction; and
outputting the determined numeric expression level of, category of, and presence of the biomarker in the region of interest.
2. The computer-implemented method of claim 1 , wherein the identifying the region of interest of the target specimen comprises determining whether a region of interest is present in any of the plurality of tiles.
3. The computer-implemented method of claim 1 , wherein the identifying the region of interest of the target specimen comprises:
segmenting the target electronic image based on edge and/or contrast, segmenting the target electronic image using color differences, and/or supervised determination; and
determining whether a region of interest is present in any of the plurality of tiles.
4. The computer-implemented method of claim 1 , further comprising:
determining numeric expression level of, a biomarker in a region of interest by determining a first expression score for each tile of the target electronic image, and aggregating the first expression score for each of the tiles to determine a second expression score for part of the target specimen.
5. The computer-implemented method of claim 1 , wherein the determining of the category of a biomarker in the region of interest comprises:
grouping numeric expression levels into one or more diagnostic categories, the one or more diagnostic categories representing a scale of a presence of the biomarker in the region of interest; and
identifying a diagnostic category, among the one or more diagnostic categories, to classify the biomarker in the region of interest.
6. The computer-implemented method of claim 1 , wherein the outputting the determined numeric expression level of, category of, and presence of the biomarker in the region of interest comprises displaying a visual indicator for the numeric expression level of, category of, and/or presence of the biomarker, and displaying one or more treatment options based on the numeric expression level of, category of, and presence of the biomarker.
7. The computer-implemented method of claim 1 , further comprising:
notifying an external user of the presence or absence of the biomarker in the region of interest.
8. The method of claim 1 , further including:
determining, based on the determined numeric expression level of, category of, and presence of the biomarker, one or more treatment recommendations for the patient.
9. The method of claim 1 , wherein applying a first machine learning algorithm to the target electronic image to identify one or more salient regions of the target electronic image further includes:
using machine learning visualization tools to create a detailed heatmap; and
extracting the salient regions from the heatmap.
10. A system for analyzing an image corresponding to a specimen, the system comprising:
at least one memory storing instructions; and
at least one processor configured to execute the instructions to perform operations comprising:
receiving a target electronic image corresponding to a target specimen, the target specimen comprising a tissue sample of a patient;
applying a first machine learning algorithm to the target electronic image to identify one or more salient regions of the target electronic image;
applying a second machine learning algorithm focusing on the one or more salient regions of the target electronic image to identify a region of interest of the target specimen and determine a numeric expression level of, category of, and a numeric probability that there is a presence of a biomarker in the region of interest of the target specimen, the biomarker comprising at least one from among an epithelial growth factor receptor (EGFR) biomarker and/or a DNA mismatch repair (MMR) deficiency biomarker, the second machine learning algorithm having been generated by processing a plurality of training images to predict whether a region of interest is present in the target electronic image, the training images comprising images of human tissue and/or images that are algorithmically generated; wherein determining the numeric expression level of the biomarker includes:
breaking the one or more salient regions of target electronic image into a plurality of tiles;
determining, by the second machine learning algorithm, the numeric expression level corresponding to a prediction for each tile to determine a plurality of tile predictions; and
aggregating the plurality of tile predictions into at least one part level numeric expression level prediction for sub-regions of the one or more salient regions, wherein the at least one part level numeric expression level prediction and prediction for each tile are fed into the second machine learning algorithm to determine the numeric expression level prediction; and
outputting the determined numeric expression level of, category of, and presence of the biomarker in the region of interest.
11. The system of claim 10 , wherein the identifying the region of interest of the target specimen comprises determining whether a region of interest is present in any of the plurality of tiles.
12. The system of claim 10 , wherein the identifying the region of interest of the target specimen comprises:
segmenting the target electronic image based on edge and/or contrast, segmenting the target electronic image using color differences, and/or supervised determination; and
determining whether a region of interest is present in any of the plurality of tiles.
13. The system of claim 10 , further comprising:
determining a numeric expression level of, a biomarker in a region of interest by determining a first expression score for each tile of the target electronic image, and aggregating the first expression score for each of the tiles to determine a second expression score for part of the target specimen.
14. The system of claim 10 , wherein the determining of the category of the biomarker in the region of interest comprises:
grouping numeric expression levels into one or more diagnostic categories, the one or more diagnostic categories representing a scale of a presence of the biomarker in the region of interest; and
identifying a diagnostic category, among the one or more diagnostic categories, to classify the biomarker in the region of interest.
15. The system of claim 10 , wherein the outputting the determined numeric expression level of, category of, and presence of the biomarker in the region of interest comprises displaying a visual indicator for the numeric expression level of, category of, and/or presence of the biomarker, and displaying one or more treatment options based on the numeric expression level of, category of, and presence of the biomarker.
16. A non-transitory computer-readable medium storing instructions that, when executed by processor, cause the processor to perform a method for analyzing an image corresponding to a specimen, the method comprising:
receiving a target electronic image corresponding to a target specimen, the target specimen comprising a tissue sample of a patient;
applying a first machine learning algorithm to the target electronic image to identify one or more salient regions of the target electronic image;
applying a second machine learning algorithm focusing on the one or more salient regions of the target electronic image to identify a region of interest of the target specimen and determine a numeric expression level of, category of, and a numeric probability that there is a presence of a biomarker in the region of interest of the target specimen, the biomarker comprising at least one from among an epithelial growth factor receptor (EGFR) biomarker and/or a DNA mismatch repair (MMR) deficiency biomarker, the second machine learning algorithm having been generated by processing a plurality of training images to predict whether a region of interest is present in the target electronic image, the training images comprising images of human tissue and/or images that are algorithmically generated; wherein determining the numeric expression level of the biomarker includes:
breaking the one or more salient regions of target electronic image into a plurality of tiles;
determining, by the second machine learning algorithm, the numeric expression level corresponding to a prediction for each tile to determine a plurality of tile predictions; and
aggregating the plurality of tile predictions into at least one part level numeric expression level prediction for sub-regions of the one or more salient regions, wherein the at least one part level numeric expression level prediction and prediction for each tile are fed into the second machine learning algorithm to determine the numeric expression level prediction; and
outputting the determined numeric expression level of, category of, and presence of the biomarker in the region of interest.
17. The non-transitory computer-readable medium of claim 16 , wherein the identifying the region of interest of the target specimen comprises determining whether a region of interest is present in any of the plurality of tiles.
18. The non-transitory computer-readable medium of claim 16 , wherein the identifying the region of interest of the target specimen comprises:
segmenting the target electronic image based on edge and/or contrast, segmenting the target electronic image using color differences, and/or supervised determination; and
determining whether a region of interest is present in any of the plurality of tiles.
19. The non-transitory computer-readable medium of claim 16 , further comprising:
determining numeric expression level of a biomarker in a region of interest by determining a first expression score for each tile of the target electronic image, and aggregating the first expression score for each of the tiles to determine a second expression score for part of the target specimen.
20. The non-transitory computer-readable medium of claim 16 , wherein the determining of the category of a biomarker in the region of interest comprises:
grouping numeric expression levels into one or more diagnostic categories, the one or more diagnostic categories representing a scale of a presence of the biomarker in the region of interest; and
identifying a diagnostic category, among the one or more diagnostic categories, to classify the biomarker in the region of interest.