IP Library › Granted Patent US 12,661,005
Granted Patent B2
US 12,661,005 · App. 18/646,646 · Granted Jun 23, 2026

Methods and systems for biomarker identification and discovery

Inventors: Seyed Mohammadmohsen Hejrati (Stanford, CA); Heming Yao (Ann Arbor, MI); Miao Zhang (Foster City, CA)
Assignee: Genentech, Inc.
A61B3/1225A61B3/0025A61B3/102A61B5/7267G16H50/30A61B2576/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,661,005
App. No.
18/646,646
Filed
Apr 25, 2024
Granted
Jun 23, 2026
Kind
B2
Art Unit
3797
USPC
600/408
Abstract

Systems and methods for determining the health status of a retina of a subject. An optical coherence tomography (OCT) volume image of a retina of a subject may be received. A health indication output is generated, via a deep learning model, using the OCT volume image. The health indication output indicates a level of association between the OCT volume image and a selected health status category for the retina. A map output for the deep learning model is generated using a saliency mapping algorithm, generating a map output for the deep learning model using a saliency mapping algorithm. The map output indicates a level of contribution of a set of regions in the OCT volume image to the health indication output generated by the deep learning model.

Claims (53)

1 . A method, comprising:

receiving an optical coherence tomography (OCT) volume image of a retina of a subject;

generating, via a deep learning model, a health indication output using the OCT volume image in which the health indication output indicates a level of association between the OCT volume image and a selected health status category for the retina;

generating a map output for the deep learning model using a saliency mapping algorithm, wherein the map output indicates a level of contribution of a set of regions in the OCT volume image to the health indication output generated by the deep learning model;

identifying a set of biomarkers in the OCT volume image for the selected health status category using the map output, comprising:

identifying, using a bounding shape, a potential biomarker region in association with a region of the set of regions indicated as being associated with the selected health status category;

generating a scoring metric for the potential biomarker region; and

identifying the potential biomarker region as including at least one biomarker for the selected health status category when the scoring metric meets a selected threshold; and

generating a biomarker map, wherein the biomarker map comprises the bounding shape and the scoring metric overlaid on the OCT volume image.

2 . The method of claim 1 , wherein the selected health status category is a selected stage of age-related macular degeneration and wherein the health indication output is a probability that the OCT volume image evidences the selected stage of age-related macular degeneration.

3 . The method of claim 2 , wherein the selected stage of age-related macular degeneration includes nascent geographic atrophy.

4 . The method of claim 1 , wherein the selected health status category represents either a current health status with respect to a time at which the OCT volume image was generated or a future health status predicted to develop within a selected period of time after the time at which the OCT volume image was generated.

5 . The method of claim 1 , wherein the saliency mapping algorithm comprises a gradient-weighted class activation mapping (Grad-CAM) algorithm and wherein the map output visually indicates the level of contribution of the set of regions in the OCT volume image to the health indication output generated by the deep learning model.

6 . The method of claim 1 , wherein the OCT volume image comprises a plurality of OCT slice images that are two-dimensional and further comprising:

generating an evaluation recommendation based on at least one of the health indication output or the map output, wherein the evaluation recommendation identifies a subset of the plurality of OCT slice images for further review, the subset including fewer than 5% of the plurality of OCT slice images.

7 . The method of claim 1 , wherein the scoring metric comprises at least one of a size of the potential biomarker region or a confidence score for the potential biomarker region.

8 . The method of claim 1 , wherein generating the map output comprises:

generating a saliency map for an OCT slice image of the OCT volume image using the saliency mapping algorithm, the saliency map indicating a degree of importance of each pixel in the OCT slice image for the selected health status category;

filtering the saliency map to generate a modified saliency map; and

overlaying the modified saliency map on the OCT slice image to generate the map output.

9 . The method of claim 1 , further comprising:

generating a treatment recommendation for the retina based on the health indication output.

10 . The method of claim 1 , wherein generating, via the deep learning model, the health indication output comprises:

generating an initial output for each OCT slice image of a plurality of OCT slice images that form the OCT volume image to form a plurality of initial outputs; and

averaging the plurality of initial outputs to form the health indication output.

11 . A method comprising:

receiving an optical coherence tomography (OCT) volume image of a retina of a subject;

generating, via a deep learning model, a health indication output using the OCT volume image in which the health indication output indicates a level of association between the OCT volume image and a selected health status category for the retina; and

generating a saliency volume map for the OCT volume image using a saliency mapping algorithm, wherein the saliency volume map indicates a level of contribution of a set of regions in the OCT volume image to the health indication output generated by the deep learning model;

detecting a set of biomarkers in the OCT volume image for the selected health status category using the saliency volume map, comprising:

identifying, using a bounding shape, a potential biomarker region in association with a region of the set of regions indicated as being associated with the selected health status category;

generating a scoring metric for the potential biomarker region; and

identifying the potential biomarker region as including at least one biomarker for the selected health status category when the scoring metric meets a selected threshold; and

generating a biomarker map, wherein the biomarker map comprises the bounding shape overlaid on the OCT volume image.

12 . The method of claim 11 , wherein the selected health status category is a selected stage of age-related macular degeneration and wherein the health indication output is a probability that the OCT volume image evidences the selected stage of age-related macular degeneration.

13 . A system, comprising:

a non-transitory memory; and

a hardware processor coupled with the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to:

receive an optical coherence tomography (OCT) volume image of a retina of a subject;

generate, via a deep learning model, a health indication output using the OCT volume image in which the health indication output indicates a level of association between the OCT volume image and a selected health status category for the retina;

generate a map output for the deep learning model using a saliency mapping algorithm, wherein the map output indicates a level of contribution of a set of regions in the OCT volume image to the health indication output generated by the deep learning model;

identify a set of biomarkers in the OCT volume image for the selected health status category using the map output, comprising:

identify, using a bounding shape, a potential biomarker region in association with a region of the set of regions indicated as being associated with the selected health status category;

generate a scoring metric for the potential biomarker region; and

identify the potential biomarker region as including at least one biomarker for the selected health status category when the scoring metric meets a selected threshold; and

generate a biomarker map, wherein the biomarker map comprises the bounding shape and the scoring metric overlaid on the OCT volume image.

14 . The system of claim 13 , wherein the selected health status category is a selected stage of age-related macular degeneration and wherein the health indication output is a probability that the OCT volume image evidences the selected stage of age-related macular degeneration.

15 . The system of claim 13 , wherein the selected health status category represents either a current health status with respect to a time at which the OCT volume image was generated or a future health status predicted to develop within a selected period of time after the time at which the OCT volume image was generated.

16 . The system of claim 13 , wherein the saliency mapping algorithm comprises a gradient-weighted class activation mapping (Grad-CAM) algorithm and wherein the map output visually indicates the level of contribution of the set of regions in the OCT volume image to the health indication output generated by the deep learning model.

17 . The system of claim 13 , wherein the OCT volume image comprises a plurality of OCT slice images that are two-dimensional and wherein the hardware processor is further configured to read instructions from the non-transitory memory to cause the system to generate an evaluation recommendation based on at least one of the health indication output or the map output, wherein the evaluation recommendation identifies a subset of the plurality of OCT slice images for further review, the subset including fewer than 5% of the plurality of OCT slice images.

18 . The method of claim 7 ,

wherein the scoring metric comprises the size of the potential biomarker region; and

wherein the size of the potential biomarker region is defined by a number of pixels.

Continuity (3)
Continuation PCTUS2022047944 · Oct 26, 2022
Provisional Application 63272060 · Oct 26, 2021
Related Publication 20240293024A1 · Sep 5, 2024
References Cited (62)
US 10074038B2 · Hsieh · 2018 [cited by examiner]
US 10354171B2 · Hsieh · 2019 [cited by examiner]
US 20170154139A1 · Iorio · 2017 [cited by examiner]
US 20180060512A1 · Sorenson · 2018 [cited by examiner]
US 20180373980A1 · Huval · 2018 [cited by examiner]
US 20190340753A1 · Brestel · 2019 [cited by examiner]
US 20200191553A1 · Rothberg · 2020 [cited by examiner]
US 20200302238A1 · Nguyen · 2020 [cited by examiner]
US 20210073986A1 · Kapur · 2021 [cited by examiner]
US 20210369195A1 · Russakoff · 2021 [cited by examiner]
US 20240293024A1 · Hejrati · 2024 [cited by examiner]
KR 102198395B1 · 2021 [cited by applicant]
WO 2019210079A1 · 2019 [cited by applicant]
Selvaraju et al., “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization,” Arxiv: 1610.02391, Mar. 21, 2017. [cited by applicant]
International Search Report and Written Opinion mailed Feb. 8, 2023 in International Application No. PCT/US2022/047944, 10 pages. [cited by applicant]
Wen et al., “Interpretable Automated Diagnosis of Retinal Disease using Deep OCT Analysis ”, https://arxiv.org/ abs/2109.02436v1, Sep. 3, 2021. [cited by applicant]
Ahnaf et al., “Understanding CNN's Decision Making on OCT-based AMD Detection”, 2021 International Conference on Electronics, Communications and Information Technology (ICECIT), IEEE, Sep. 14-16, 2021, pp. 1-4. [cited by applicant]
International Preliminary Report on Patentability issued Apr. 30, 2024 in International Application No. PCT/US2022/047944, 7 pages. [cited by applicant]
International Search Report and Written Opinion mailed Aug. 2, 2023 in International Application No. PCT/US2023/021420, 12 pages. [cited by applicant]
George et al., “3D-CNN for Glaucoma Detection Using Optical Coherence Tomography”, 16th European Conference—Computer Vision—ECCV 2020, vol. 7, Dec. 31, 2019, pp. 52-59. [cited by applicant]
International Preliminary Report on Patentability issued Nov. 7, 2024 in International Application No. PCT/US2023/021420, 8 pages. [cited by applicant]
Wu et al., “Optical Coherence Tomography-Defined Changes Preceding the Development of Drusen-Associated Atrophy in Age-Related Macular Degeneration,” American Academy of Ophthalmology, Ophthalmology, Dec. 2014, vol. 121… [cited by applicant]
Wu et al., “Prospective Longitudinal Evaluation of Nascent Geographic Atrophy in Age-Related Macular Degeneration,” Jun. 2020, American Academy of Ophthalmology, Ophthalmology Retina, vol. 4, No. 6, pp. 568-575. [cited by applicant]
Wu et al., “Can the Onset of Atrophic Age-Related Macular Degeneration Be an Acceptable Endpoint for Preventative Trials?” Ophthalmologica, International journal of ophthalmology, Dec. 2020, vol. 243, Issue 6, pp. 399-4… [cited by applicant]
Derradji et al., “Fully-automated atrophy segmentation in dry age-related macular degeneration in optical coherence tomography.” Nov. 8, 2021, Scientific Reports, 11(1), Article 21893. [cited by applicant]
Corradetti et al., “Automated Identification of Incomplete and Complete Retinal Epithelial Pigment and Outer Retinal Atrophy Using Machine Learning.” Investigative Ophthalmology and Visual Science, Jun. 2022, vol. 63, N… [cited by applicant]
Chiang et al., “Automated Identification of Incomplete and Complete Retinal Epithelial Pigment and Outer Retinal Atrophy Using Machine Learning.” American Academy of Ophthalmology, Ophthalmology Retina, Feb. 2023, vol. … [cited by applicant]
Yang et al., “Weakly supervised lesion localization for age-related macular degeneration detection using optical coherence tomography images.” PLOS ONE, Apr. 5, 2019, 14(4);e0215076. [cited by applicant]
Selvaraju et al., “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.” International Journal of Computer Vision, 2020, vol. 128, pp. 336-359. [cited by applicant]
Shi et al., “Improving Interpretability in Machine Diagnosis: Detection of Geographic Atrophy in OCT Scans,” American Academy of Ophthalmology, Ophthalmology Science, Sep. 2021, vol. 1, No. 3, 9 pages. [cited by applicant]
Yoon et al., “Optical coherence tomography-based deep-learning model for detecting central serous chorioretinopathy.” Scientific reports, 2020, vol. 10(1), 18852, 9 pages. [cited by applicant]
Wang et al., “Explainable Deep Learning for Biomarker Classification of OCT Images.” 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), Cincinnati, Ohio, pp. 204-210. [cited by applicant]
Li et al., “Development and validation of a deep learning system to screen vision-threatening conditions in high myopia using optical coherence tomography images.” 2022, The British Journal of Ophthalmology, vol. 106, N… [cited by applicant]
Guymer et al., “Subthreshold Nanosecond Laser Intervention in Age-Related Macular Degeneration: The LEAD Randomized Controlled Clinical Trial.” Jun. 2019, American Academy of Ophthalmology, Ophthalmology, vol. 126, No. … [cited by applicant]
Wu et al., “Predicting progression of age-related macular degeneration using optical coherence tomography and fundus photography.” Feb. 2021, Ophthalmology Retina, vol. 5, No. 2, pp. 118-125. [cited by applicant]
Guymer et al., “Incomplete Retinal Pigment Epithelial and Outer Retinal Atrophy in Age-Related Macular Degeneration: Classification of Atrophy Meeting Report 4.” Mar. 2021, Ophthalmology, vol. 127, No. 3, pp. 394-409. [cited by applicant]
Carbonneau et al., “Multiple instance learning: A survey of problem characteristics and applications.” May 2018, Pattern Recognition, vol. 77, pp. 329-353. [cited by applicant]
Otsu et al., “A Threshold Selection Method from Gray-Level Histograms.” Jan. 1979, IEEE Transactions on Systems, Man, and Cybernetics, vol. SMC-9, No. 1, pp. 62-66. [cited by applicant]
Liu et al., “A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.” Oct. 2019, The Lancet Digital Health, vol. 1, … [cited by applicant]
Redmon et al., “YOLOv3: An Incremental Improvement.” Apr. 8, 2018. arXiv preprint arXiv:1804.02767. [cited by applicant]
Holz et al., “Geographic atrophy: Clinical features and potential therapeutic approaches.” May 2014. Ophthalmology, vol. 121, No. 5, pp. 1079-1091. [cited by applicant]
Areds, “A randomized, placebo-controlled, clinical trial of high-dose supplementation with vitamins C and E, beta carotene and zinc for age-related macular degeneration and vision loss. Areds Report No. 8.” Archives of … [cited by applicant]
AREDS 2, “Lutein + Zeaxanthin and Omega-3 Fatty Acids for Age-Related Macular Degeneration.” May 15, 2013, JAMA, vol. 309, No. 19, pp. 2005-2015. [cited by applicant]
Lazaridis et al., “OCT Signal Enhancement with Deep Learning.” May-Jun. 2021. Ophthalmology Glaucoma, vol. 4, Issue 3, pp. 295-304. [cited by applicant]
Wu et al., “Fundus autofluorescence characteristics of nascent geographic atrophy in age-related macular degeneration.” Mar. 2015, Retina, Investigative Ophthalmology & Visual Science, vol. 56, No. 3, pp. 1546-1552. [cited by applicant]
Wu et al., “Microperimetry of nascent geographic atrophy in age-related macular degeneration.” Jan. 2015. Investigative Ophthalmology & Vision Science, vol. 56, No. 1, pp. 115-121. [cited by applicant]
Ferrara et al., “Optical coherence tomography features preceding the onset of advanced age-related macular degeneration.” Jul. 2017. Investigative Ophthalmology & Vision Science, vol. 58, No. 9, pp. 3519-3529. [cited by applicant]
Ting et al., “Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes.” Dec. 12, 2017. JAMA, vol. 318, No. 2… [cited by applicant]
Bellemo et al., “Artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: A clinical validation study.” May 2019. The Lancet Digital Health, vol. 1, pp. … [cited by applicant]
Abramoff et al., “Pivotal trial of an autonomous ai-based diagnostic system for detection of diabetic retinopathy in primary care offices.” Aug. 28, 2018. npj Digital Medicine, vol. 1, No. 1, Article 39, 8 pages. [cited by applicant]
Ruamviboonsuk et al., “Real-time diabetic retinopathy screening by deep learning in a multisite national screening programme: A prospective interventional cohort study.” Apr. 2022. The Lancet Digital Health, vol. 4, pp.… [cited by applicant]
Keenan et al., “A deep learning approach for automated detection of geographic atrophy from color fundus photographs.” Nov. 2019. Ophthalmology, vol. 126, Issue 11, pp. 1533-1540. [cited by applicant]
Goh et al., “Hyporeflective cores within drusen: Association with progression of age-related macular degeneration and impact on visual sensitivity.” Apr. 2022. Ophthalmology Retina, vol. 6, No. 4, pp. 284-290. [cited by applicant]
Goh et al., “Cuticular drusen in age-related macular degeneration: Association with progression and impact on visual sensitivity.” Jun. 2022. Ophthalmology, vol. 129, No. 6, pp. 653-660. [cited by applicant]
Wu et al., “Secondary and exploratory outcomes of the subthreshold nanosecond laser intervention randomized trial in age-related macular degeneration: A Lead study report.” Dec. 2019. Ophthalmology Retina, vol. 3, No. 1… [cited by applicant]
Ferris et al., “Clinical classification of age-related macular degeneration.” Apr. 2013. Ophthalmology, vol. 120, Issue 4, pp. 844-851. [cited by applicant]
Sadda et al., “Consensus definition for atrophy associated with age-related macular degeneration on OCT: Classification of Atrophy Report 3.” Apr. 2018. Ophthalmology, vol. 125, Issue 4, pp. 537-548. [cited by applicant]
Davis et al., “The age-related eye disease study severity scale for age-related macular degeneration: AREDS Report No. 17.” Nov. 2005. Archives of Ophthalmology, vol. 123, No. 11, pp. 1484-1498. [cited by applicant]
Kumar et al., “A clinical perspective on the expanding role of artificial intelligence in age-related macular degeneration.” Jan. 24, 2022. Clinical and Experimental Optometry, vol. 105, No. 7, pp. 674-679. [cited by applicant]
Wu et al., “Incomplete retinal pigment epithelial and outer retinal atrophy: longitudinal evaluation in age-related macular degeneration.” Feb. 2023. Ophthalmology, vol. 130, No. 2, pp. 205-212. [cited by applicant]
Lee et al., “How Artificial Intelligence Can Transform Randomized Controlled Trials.” 2020. Translational Vision Science & Technology, vol. 9, No. 2, Article 9, 4 pages. [cited by applicant]
Loo et al., “Beyond Performance Metrics: Automatic Deep Learning Retinal OCT Analysis Reproduces Clinical Trial Outcome.” Jun. 2020. Ophthalmology, vol. 127, Issue 6, pp. 793-801. [cited by applicant]