IP Library Granted Patent US 12,731,687
Granted Patent B2
US 12,731,687 · App. 17/747,536 · Granted Sep 8, 2026

Systems and methods for machine learning (ML) model diagnostic assessments based on digital pathology data

Inventors: Benjamin Glass (Boston, MA); Surya Teja Chavali (Boston, MA); Syed Ashar Javed (Boston, MA); Shamira Sridharan Weaver (Boston, MA); Murray Resnick (Boston, MA); Ilan Wapinski (Brookline, MA); Michael Montalto (Boston, MA); Andrew Hanno Beck (Brookline, MA); Aditya Khosla (Watertown, MA)
Assignee: PATHAI, INC.
G16H50/20G06T7/0012G06V10/25G16H70/60G06T2207/30096
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,731,687
App. No.
17/747,536
Filed
May 18, 2022
Granted
Sep 8, 2026
Kind
B2
Art Unit
2661
USPC
382/170
Abstract

Techniques for performing diagnostic assessments based on digital pathology data are disclosed. In one particular embodiment, the techniques may be realized as a method for performing a diagnostic assessment based on digital pathology data comprising obtaining first digital pathology data comprising intensity information, the first digital pathology data being associated with a plurality of regions of interest in a biological sample; applying first machine learning models to the first digital pathology data, the first machine learning models identifying first regions of interest among the plurality of regions of interest based on the intensity information; applying second machine learning models to the first digital pathology data, the second machine learning models identifying at least one pattern associated with at least one of the first regions of interest; generating a diagnostic assessment based on the first regions of interest and the at least one pattern.

Claims (57)

1 . A method for performing a diagnostic assessment based on digital pathology data, comprising:

obtaining first digital pathology data comprising intensity information, the first digital pathology data being associated with a plurality of regions of interest in a biological sample;

applying one or more first machine learning models to the first digital pathology data, the one or more first machine learning models identifying one or more first regions of interest among the plurality of regions of interest based on the intensity information;

applying one or more second machine learning models to the first digital pathology data, the one or more second machine learning models identifying at least one pattern associated with at least one of the one or more first regions of interest;

generating a diagnostic assessment based on the one or more first regions of interest and the at least one pattern; and

detecting drift in diagnostic assessments performed by at least one pathologist in a clinical trial by comparing the generated diagnostic assessment with the pathologist diagnostic assessments, wherein detecting drift comprises detecting a change over time in at least one of nomenclature, grading of lesions, and scoring of a bio marker.

2 . The method of claim 1 , wherein the first digital pathology data comprises one or more whole slide images.

3 . The method of claim 2 , wherein the one or more whole slide images corresponds to a tumor biopsy sample stained using anti-HER2 immunohistochemistry.

4 . The method of claim 3 , wherein the diagnostic assessment comprises a HER2 score.

5 . The method of claim 1 , wherein the one or more first regions of interest comprises at least one of a tissue region of interest or a cell of interest.

6 . The method of claim 5 , wherein the at least one of the tissue region of interest or the cell of interest comprises one or more of a cancer epithelium, a cancer stroma, a ductal carcinoma in situ, a necrosis, a cell membrane, or an artifact.

7 . The method of claim 1 , wherein the least one pattern comprises a staining pattern of a cell membrane.

8 . The method of claim 7 , wherein the staining pattern is selected from a group consisting of: negative or unstained, partial positive, and complete positive.

9 . The method of claim 1 , wherein the diagnostic assessment comprises a precision slide-level score.

10 . The method of claim 1 , wherein the diagnostic assessment comprises an adjusted slide level score, the adjusted slide level score being generated using machine learning model predictions optimized for consensus between the adjusted slide level score and a slide level score provided by a pathologist.

11 . The method of claim 1 , further comprising applying one or more third machine learning models to the first digital pathology data, the one or more third machine learning models identifying an intensity associated with at least one of the one or more first regions of interest.

12 . The method of claim 11 , wherein the intensity corresponds to an intensity of staining of cell membranes, wherein the intensity is selected from a group consisting of: unstained, faintly stained, moderately stained, or completely stained.

13 . The method of claim 1 , further comprising extracting one or more histological features associated with the first digital pathology data.

14 . The method of claim 1 , further comprising calculating one or more cell-level features associated with the first digital pathology data.

15 . The method of claim 14 , wherein the one or more cell-level features are based on a number of cells corresponding to each American Society of Clinical Oncology/College of American Pathologists (ASCO/CAP) category identified in the first digital pathology data.

16 . The method of claim 1 , wherein the tumor biopsy sample is derived from a patient with breast cancer.

17 . A system for performing a diagnostic assessment based on digital pathology data comprising:

at least one computer processor, wherein the at least one computer processor is configured to:

obtain first digital pathology data comprising intensity information, the first digital pathology data being associated with a plurality of regions of interest in a biological sample;

apply one or more first machine learning models to the first digital pathology data, the one or more first machine learning models identifying one or more first regions of interest among the plurality of regions of interest based on the intensity information;

apply one or more second machine learning models to the first digital pathology data, the one or more second machine learning models identifying at least one pattern associated with at least one of the one or more first regions of interest;

generate a diagnostic assessment based on the one or more first regions of interest and the at least one pattern; and

detect drift in diagnostic assessments performed by at least one pathologist in a clinical trial by comparing the generated diagnostic assessment with the pathologist diagnostic assessments, wherein detecting drift comprises detecting a change over time in at least one of nomenclature, grading of lesions, and scoring of a bio marker.

18 . The system of claim 17 , wherein the first digital pathology data comprises one or more whole slide images, the one or more whole slide images corresponding to a tumor biopsy sample stained using anti-HER2 immunohistochemistry, and wherein the diagnostic assessment comprises a HER2 score.

19 . The system of claim 17 , wherein the one or more first regions of interest comprises at least one of a tissue region of interest or a cell of interest, the at least one of the tissue region of interest or the cell of interest comprising one or more of a cancer epithelium, a cancer stroma, a ductal carcinoma in situ, a necrosis, a cell membrane, or an artifact.

20 . The system of claim 17 , wherein the least one pattern comprises a staining pattern of a cell membrane, the staining pattern being selected from a group consisting of: negative or unstained, partial positive, and complete positive.

21 . The system of claim 17 , wherein the diagnostic assessment comprises:

a precision slide-level score; and

an adjusted slide level score, the adjusted slide level score being generated using machine learning model predictions optimized for consensus between the adjusted slide level score and a slide level score provided by a pathologist.

22 . The system of claim 17 , further comprising:

applying one or more third machine learning models to the first digital pathology data, the one or more third machine learning models identifying an intensity associated with at least one of the one or more first regions of interest, wherein the intensity corresponds to an intensity of staining of cell membranes, wherein the intensity is selected from a group consisting of: unstained, faintly stained, moderately stained, or completely stained;

extracting one or more histological features associated with the first digital pathology data; and

calculating one or more cell-level features associated with the first digital pathology data.

23 . An article of manufacture for performing a diagnostic assessment based on digital pathology data comprising:

a non-transitory processor readable medium; and

instructions stored on the medium;

wherein the instructions are configured to be readable from the medium by at least one computer processor and thereby cause the at least one computer processor to operate so as to:

obtain first digital pathology data comprising intensity information, the first digital pathology data being associated with a plurality of regions of interest in a biological sample;

apply one or more first machine learning models to the first digital pathology data, the one or more first machine learning models identifying one or more first regions of interest among the plurality of regions of interest based on the intensity information;

apply one or more second machine learning models to the first digital pathology data, the one or more second machine learning models identifying at least one pattern associated with at least one of the one or more first regions of interest;

generate a diagnostic assessment based on the one or more first regions of interest and the at least one pattern; and

detect drift in diagnostic assessments performed by at least one pathologist in a clinical trial by comparing the generated diagnostic assessment with the pathologist diagnostic assessments, wherein detecting drift comprises detecting a change over time in at least one of nomenclature, grading of lesions, and scoring of a bio marker.

24 . The article of manufacture of claim 23 , wherein the first digital pathology data comprises one or more whole slide images, the one or more whole slide images corresponding to a tumor biopsy sample stained using anti-HER2 immunohistochemistry, and wherein the diagnostic assessment comprises a HER2 score.

25 . The article of manufacture of claim 23 , wherein the one or more first regions of interest comprises at least one of a tissue region of interest or a cell of interest, the at least one of the tissue region of interest or the cell of interest comprising one or more of a cancer epithelium, a cancer stroma, a ductal carcinoma in situ, a necrosis, a cell membrane, or an artifact.

26 . The article of manufacture of claim 23 , wherein the least one pattern comprises a staining pattern of a cell membrane, the staining pattern being selected from a group consisting of: negative or unstained, partial positive, and complete positive.

27 . The article of manufacture of claim 23 , wherein the diagnostic assessment comprises:

a precision slide-level score; and

an adjusted slide level score, the adjusted slide level score being generated using machine learning model predictions optimized for consensus between the adjusted slide level score and a slide level score provided by a pathologist.

28 . The article of manufacture of claim 23 , further comprising:

applying one or more third machine learning models to the first digital pathology data, the one or more third machine learning models identifying an intensity associated with at least one of the one or more first regions of interest, wherein the intensity corresponds to an intensity of staining of cell membranes, wherein the intensity is selected from a group consisting of: unstained, faintly stained, moderately stained, or completely stained;

extracting one or more histological features associated with the first digital pathology data; and

calculating one or more cell-level features associated with the first digital pathology data.

Assignments (5)
RELEASE OF SECURITY INTEREST Recorded Jul 28, 2026
From: ORBIMED ROYALTY & CREDIT OPPORTUNITIES IV, LP, AS ADMINISTRATIVE AGENT
To: PATHAI, INC.
Reel/Frame 075427/0613 →
SECURITY INTEREST Recorded Sep 22, 2025
From: PATHAI, INC.
To: ORBIMED ROYALTY & CREDIT OPPORTUNITIES IV, LP, AS ADMINISTRATIVE AGENT FOR SECURED PARTIES
Reel/Frame 072322/0631 →
RELEASE OF SECURITY INTEREST Recorded Sep 18, 2025
From: HERCULES CAPITAL, INC., AS AGENT
To: PATHAI, INC.
Reel/Frame 072300/0731 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 5, 2025
From: GLASS, BENJAMIN; CHAVALI, SURYA TEJA; JAVED, SYED ASHAR; WEAVER, SHAMIRA SRIDHARAN; RESNICK, MURRAY; WAPINSKI, ILAN; BECK, ANDREW HANNO; MONTALTO, MICHAEL; KHOSLA, ADITYA
To: PATHAI, INC.
Reel/Frame 071023/0822 →
SECURITY INTEREST Recorded Dec 23, 2022
From: PATHAI, INC.
To: HERCULES CAPITAL, INC., AS AGENT
Reel/Frame 062195/0001 →
Continuity (2)
Provisional Application 63190162 · May 18, 2021
Related Publication 20220375606A1 · Nov 24, 2022
References Cited (49)
US 10650520B1 · Beck et al. · 2020 [cited by applicant]
US 10957041B2 · Yip et al. · 2021 [cited by applicant]
US 10991097B2 · Yip et al. · 2021 [cited by applicant]
US 11640859B2 · Colley et al. · 2023 [cited by applicant]
US 12112839B2 · Colley et al. · 2024 [cited by applicant]
US 12175666B2 · Wang et al. · 2024 [cited by applicant]
US 20120076390A1 · Potts · 2012 [cited by examiner]
US 20150110381A1 · Parvin et al. · 2015 [cited by applicant]
US 20160027226A1 · Gigl et al. · 2016 [cited by applicant]
US 20160110584A1 · Remiszewski et al. · 2016 [cited by applicant]
US 20160267226A1 · Xu · 2016 [cited by examiner]
US 20170212018A1 · Grunkin · 2017 [cited by examiner]
US 20180232883A1 · Sethi et al. · 2018 [cited by applicant]
US 20200150258A1 · Itkin · 2020 [cited by applicant]
US 20200258223A1 · Yip · 2020 [cited by applicant]
US 20200294231A1 · Tosun et al. · 2020 [cited by applicant]
US 20200342597A1 · Chukka et al. · 2020 [cited by applicant]
US 20210073986A1 · Kapur · 2021 [cited by examiner]
US 20210093249A1 · Anand · 2021 [cited by examiner]
US 20220245802A1 · Wang et al. · 2022 [cited by applicant]
US 20220375606A1 · Glass · 2022 [cited by applicant]
EP 3576096A1 · 2019 [cited by applicant]
WO WO2020150258A1 · 2020 [cited by applicant]
WO WO2022038527A1 · 2022 [cited by applicant]
WO WO2022165433A1 · 2022 [cited by applicant]
WO WO2022245925A1 · 2022 [cited by applicant]
Robinson, Max, et al. “Quality assurance guidance for scoring and reporting for pathologists and laboratories undertaking clinical trial work.” The Journal of Pathology: Clinical Research 5.2 (2019): 91-99.https://paths… [cited by examiner]
International Search Report and Written Opinion issued by the U.S. Patent and Trademark Office as International Searching Authority in Interntaional Application No. PCT/US22/29807, dated Sep. 23, 2022 (10 pages). [cited by applicant]
Glass, et al., “Abstract #3061: Machine Learning Models to Quantify HER2 for Real-Time Tissue Image Analysis in Prospective Clinical Trials”, American Society of Clinical Oncology Annual Meeting, Virtual Meeting, Jun. 4… [cited by applicant]
Marchio, et al., “Evolving concepts in HER2 evaluation in breast cancer: Heterogeneity, HER2-low carcinomas and beyond”, Seminars in Cancer Biology, 72:123-135, available online Feb. 26, 2020 (13 pages). [cited by applicant]
Modi, et al., “Trastuzumab Deruxtecan in Previously Treated HER2-Positive Breast Cancer”, New England Journal of Medicine, 382(7):610-621, 2020, available Dec. 11, 2019 (12 pages). [cited by applicant]
Perez, et al., “HER2 Testing by Local, Central, and Reference Laboratories in Specimens From the North Central Cancer Treatment Group N9831 Intergroup Adjuvant Trial”, Journal of Clinical Oncology, 24(19):3032-3038, Jul… [cited by applicant]
Adnan, et al., “Representation Learning of Histopathology Images using Graph Neural Networks”, IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4254-4261, 2020 (8 pages). [cited by applicant]
Diehl, “Edge Contraction Pooling for Graph Neural Networks”, https://arxiv.org/pdf/1905.10990, May 27, 2019 (9 pages). [cited by applicant]
European Extended Search Report issued in European Application No. EP22746881.6, dated Oct. 24, 2024 (9 pages). [cited by applicant]
International Search Report and Written Opinion issued by the U.S. Patent and Trademark Office in International Application No. PCT/US22/14778, dated May 12, 2022 (11 pages). [cited by applicant]
Krizhevsky, et al., “ImageNet Classification with Deep Convolutional Neural Networks”, original version published in Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe,… [cited by applicant]
Lee, et al., “Self-Attention Graph Pooling”, Proceedings of the 36th International Conference on Machine Learning, Long Beach, California, PMLR97, https://arXiv:1904.08082v4, Jun. 13, 2019 (10 pages). [cited by applicant]
Levy, et al., “A large-scale internal validation study of unsupervised virtual trichrome staining technologies on nonalcoholic steatohepatitis liver biopsies”, Modern Pathology, 34:808-822, 2021, published online Dec. 9… [cited by applicant]
Morris, et al., “Weisfeiler and Leman Go Neural: Higher-Order Graph Neural Networks,” Association for the Advancement of Artificial Intelligence, Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), 33(01)… [cited by applicant]
Szegedy, et al., “Going Deeper with convolutions,” 2015 IEEE Conf. on Computer Vision and Pattern Recognition, https://arXiv:1409.4842v1, Sep. 17, 2014 (12 pages). [cited by applicant]
Wang, et al., “Face Search at Scale: 80 Million Gallery”, MSU Technical Report MSU-CSE-15- 11, https://arXiv:1507.07242v2, Jul. 24, 2015 (15 pages). [cited by applicant]
Zhang, et al., “BIRCH: An Efficient Data Clustering Method for Very Large Databases,” Proc. 1996 ACM SIGMOD Intl. Conf. on Management of Data, Montreal, Canada, 103-114, 1996 (12 pages). [cited by applicant]
Khameneh, et al., “Automated segmentation of cell membranes to evaluate HER2 status in whole slide images using a modified deep learning network”, Computers in Biology and Medicine, 110:164-174, 2019 (11 pages). [cited by applicant]
Masmoudi, et al., “Automated Quantitative Assessment of HER-2/neu Immunohistochemical Expression in Breast Cancer”, IEEE Transactions on Medical Imaging, 28(6):916-925, Jun. 2009 (10 pages). [cited by applicant]
Qaiser, et al., “Learning Where to See: A Novel Attention Model for Automated Immunohistochemical Scoring”, IEEE Transactions on Medical Imaging, 38(11):2620-2631, Nov. 2019 (12 pages). [cited by applicant]
Vandenberghe, et al., “Relevance of deep learning to facilitate the diagnosis of HER2 status in breast cancer”, Scientific Reports, 7:45938, Apr. 5, 2017 (11 pages). [cited by applicant]
European Extended Search Report issued in European Patent Application EP22805386.4, dated Mar. 14, 2025 (17 pages). [cited by applicant]
Robinson, et al., “Quality assurance guidance for scoring and reporting for pathologists and laboratories undertaking clinical trial work”, Journal of Pathology: Clinical Research, 5:91-99, Apr. 2019, published online N… [cited by applicant]