Systems and methods for processing electronic images for biomarker localization
Systems and methods are disclosed for receiving digital images of a pathology specimen from a patient, the pathology specimen comprising tumor tissue, the one or more digital images being associated with data about a plurality of biomarkers in the tumor tissue and data about a surrounding invasive margin around the tumor tissue; identifying the tumor tissue and the surrounding invasive margin region to be analyzed for each of the one or more digital images; generating, using a machine learning model on the one or more digital images, at least one inference of a presence of the plurality of biomarkers in the tumor tissue and the surrounding invasive margin region; determining a spatial relationship of each of the plurality of biomarkers identified in the tumor tissue and the surrounding invasive margin region to themselves and to other cell types; and determining a prediction for a treatment outcome and/or at least one treatment recommendation for the patient.
1. A computer-implemented method, comprising:
receiving one or more digital images of a pathology specimen from a patient, the one or more digital images comprising data about a plurality of biomarkers, a tumor, and a surrounding invasive margin region;
generating, using a machine learning model on the one or more digital images, at least one inference of a presence of the plurality of biomarkers in the tumor and the surrounding invasive margin region, wherein the machine learning model is configured to compare the plurality of biomarkers and determine a spatial relationship, the spatial relationship being determined across at least a portion of the tumor and the surrounding invasive margin region using at least one spatially structured input received from a localized biomarker prediction module; and
predicting a treatment outcome, at least one resistance prediction, and/or at least one treatment recommendation for the patient.
2. The computer-implemented method of claim 1 , wherein generating the at least one inference of the presence of the plurality of biomarkers comprises receiving metadata corresponding to the at least one spatially structured input.
3. The computer-implemented method of claim 1 , wherein the at least one spatially structured input from the localized biomarker prediction module comprises a plurality of vector embeddings.
4. The computer-implemented method of claim 1 , wherein the at least one spatially structured input comprises a spatial structure location comprising information about whether locations of the plurality of biomarker lie in a tumor and/or the surrounding invasive margin region.
5. The computer-implemented method of claim 4 , further comprising:
receiving one or more training digital images associated with a training pathology specimen and an associated indication of an absence of a tumor region; and
dividing the one or more training digital images into at least one sub-region to determine if a tumor tissue is absent in the at least one sub-region.
6. The computer-implemented method of claim 1 , further comprising identifying tumor tissue and the surrounding invasive margin region for each of the one or more digital images by the machine learning model.
7. The computer-implemented method of claim 6 , further comprising:
receiving one or more training digital images associated with a training pathology specimen and an associated indication of a presence of a tumor region; and
dividing the one or more training digital images into at least one sub-region to determine if the tumor tissue is present in the at least one sub-region.
8. The computer-implemented method of claim 6 , further comprising:
receiving one or more digital images associated with a target pathology specimen and an associated indication of a presence of a tumor region;
dividing the one or more digital images into at least one sub-region to analyze to determine if the tumor tissue is present in the at least one sub-region;
applying the machine learning model to one of the one or more digital images to predict which regions of each digital image of the one or more digital images show a tumor tissue; and
indicating and flagging a location of at least one tumor region.
9. The computer-implemented method of claim 1 , wherein generating the at least one inference of the presence of the plurality of biomarkers further comprises:
receiving one or more training digital images of the pathology specimen;
receiving a plurality of data on a level of the plurality of biomarkers present in a tumor and/or an invasive margin region shown in one of the one or more training digital images;
dividing one of the one or more training digital images into at least one sub-region to determine at least one property for the at least one sub-region; and
identifying at least one tumor and/or at least one invasive margin region relevant to a biomarker of interest.
10. The computer-implemented method of claim 1 , wherein generating the at least one inference comprises determining at least one region of interest in a tumor tissue, and applying the machine learning model to determine a prediction of a biomarker expression level in the at least one region of interest.
11. The computer-implemented method of claim 1 , wherein predicting the treatment outcome includes applying the machine learning model to predict the treatment outcome from the localized biomarker.
12. A system, comprising:
at least one memory storing instructions; and
at least one processor configured to execute the instructions to perform operations comprising:
receiving one or more digital images of a pathology specimen from a patient, the one or more digital images comprising data about a plurality of biomarkers, a tumor, and a surrounding invasive margin region;
generating, using a machine learning model on the one or more digital images, at least one inference of a presence of the plurality of biomarkers in the tumor and the surrounding invasive margin region, wherein the machine learning model is configured to compare the plurality of biomarkers and determine a spatial relationship, the spatial relationship being determined across at least a portion of the tumor and the surrounding invasive margin region using at least one spatially structured input received from a localized biomarker prediction module; and
predicting a treatment outcome, at least one resistance prediction, and/or at least one treatment recommendation for the patient.
13. The system of claim 12 , wherein the pathology specimen comprises a cytology specimen.
14. The system of claim 12 , wherein the at least one spatially structured input comprises a spatial structure location comprising information about whether locations of the plurality of biomarker lie in a tumor and/or the surrounding invasive margin region.
15. The system of claim 12 , wherein the data associated with the plurality of biomarkers is identified from at least one of genetic testing, flow cytometry, and immunohistochemistry.
16. The system of claim 12 , wherein the operations further comprise identifying a tumor tissue and the surrounding invasive margin region using the machine learning model by:
receiving one or more training digital images associated with a training pathology specimen and an associated indication of a presence or an absence of a tumor region; and
dividing the one or more training digital images into at least one sub-region to determine if the tumor tissue is present in the at least one sub-region.
17. The system of claim 16 , wherein the operations further comprise applying the machine learning model to one of the one or more digital images to predict which regions show a tumor tissue or an invasive margin.
18. The system of claim 17 , wherein the operations further comprise indicating and flagging a location of at least one tumor region.
19. The system of claim 12 , wherein generating the at least one inference of the presence of the one or more biomarkers further comprises:
receiving one or more training digital images of the pathology specimen;
receiving a plurality of data on a level of the plurality of biomarkers present in a tumor and/or an invasive margin region shown in one of the one or more training digital images;
dividing one of the one or more training digital images into at least one sub-region to determine at least one property for the at least one sub-region; and
identifying at least one tumor and/or at least one invasive margin region relevant to a biomarker of interest.
20. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method comprising:
receiving one or more digital images of a pathology specimen from a patient, the one or more digital images comprising data about a plurality of biomarkers, a tumor, and a surrounding invasive margin region;
generating, using a machine learning model on the one or more digital images, at least one inference of a presence of the plurality of biomarkers in the tumor and the surrounding invasive margin region, wherein the machine learning model is configured to compare the plurality of biomarkers and determine a spatial relationship, the spatial relationship being determined across at least a portion of the tumor and the surrounding invasive margin region using at least one spatially structured input received from a localized biomarker prediction module; and
predicting a treatment outcome, at least one resistance prediction, and/or at least one treatment recommendation for the patient.