IP Library › Granted Patent US 12,499,537
Granted Patent B2
US 12,499,537 · App. 17/776,944 · Granted Dec 16, 2025

Systems and methods of deep learning for colorectal polyp screening

Inventors: Shyam S. Mohapatra (Lutz, FL); Subhra Mohapatra (Lutz, FL); Anmol Warman (Lutz, FL)
Assignee: UNIVERSITY OF SOUTH FLORIDA
G06T7/0012G06N3/045G06N3/08G06N3/044G06T2207/10068G06T2207/20084G06T2207/30032
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,499,537
App. No.
17/776,944
Granted
Dec 16, 2025
Kind
B2
Abstract

Disclosed are various embodiments of systems and methods of deep learning for colorectal polyp screening and providing a prediction of neoplasticity of a polyp. A video of a colonoscopy procedure can be captured. Frames from the video or images associated with the colonoscopy procedure can be extracted. A model for classifying objects that appear in the frames or the images can be obtained. A classification can be determined for a polyp that appears in at least one of the frames or images based on applying the frames or images to an input layer of the model.

Claims (27)

1 . A method for colorectal polyp screening,:

capturing, by a computing device, a video of a colonoscopy procedure;

extracting, by the computing device, frames from the video or images associated with the colonoscopy procedure;

generating an initial set of numerical parameter values characteristic of the extracted frames or images;

generating a reduced set of numerical parameter values by applying at least one of normalizing, unstacking, or reducing noise associated with the extracted frames or images;

generating, based at least in part on applying a nonlinear dimensionality reduction algorithm to the reduced set of numerical parameter values, reduced dimensional embeddings of a plurality of image representations of polyps that appear in the images;

obtaining, by the computing device, a model for classifying polyps that appear in the frames or images; and

determining, by the computing device, a classification of at least one polyp that appears in at least one of the frames or images based at least in part on applying the reduced set of numerical parameter values to an input layer of the model.

2 . The method of claim 1 , wherein the classification of the at least one polyp comprises an adenoma polyp, a hyperplastic polyp, or a serrated polyp.

3 . The method of claim 1 , wherein obtaining the model is based at least in part on determining weights and biases for the model.

4 . The method of claim 1 , wherein the nonlinear dimensionality reduction algorithm comprises T-distributed Stochastic Neighbor Embedding (t-SNE).

5 . The method of claim 1 , wherein generating the initial set comprises identifying each respective image as associated with a White Light (WL) diagnostic method or a Narrow Band Imaging (NBI) diagnostic method.

6 . The method of claim 1 , wherein the model is a neural network that is a multilayer perceptron or deep neural network that utilizes forward and backward propagation for training.

7 . The method of claim 6 , wherein the multilayer perceptron comprises at least: the input layer, a first hidden layer, a second hidden layer, and an output layer.

8 . A system for colorectal polyp screening, comprising:

at least one computing device; and

a non-transitory computer readable medium storing program instructions executable in the at least one computing device that, when executed by the at least one computing device, cause the at least one computing device to:

capture a video of a colonoscopy procedure;

extract frames from the video or images associated with the colonoscopy procedure;

generate an initial set of numerical parameter values characteristic of the extracted frames or images;

generate a reduced set of numerical parameter values by applying at least one of normalizing, unstacking, or reducing noise associated with the extracted frames or images;

generate, based at least in part on applying a nonlinear dimensionality reduction algorithm to the reduced set of numerical parameter values, reduced dimensional embeddings of a plurality of image representations of polyps that appear in the images;

obtain a model for classifying objects that appear in the frames or images; and

determine a classification of at least one polyp that appears in at least one of the frames based at least in part on applying the reduced set of numerical parameter values to an input layer of the model.

9 . The system of claim 8 , wherein the classification comprises an adenoma polyp, a hyperplastic polyp, or a serrated polyp.

10 . The system of claim 8 , wherein the program instructions, when executed by the at least one computing device, cause the at least one computing device to:

render, on a display of the at least one computing device, a user interface comprising an indication associated with the classification of the at least one polyp.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 27, 2022
From: MOHAPATRA, SHYAM S.; MOHAPATRA, SUBHRA; WARMAN, ANMOL
To: UNIVERSITY OF SOUTH FLORIDA
Reel/Frame 061224/0304 →
Continuity (2)
Provisional Application 62934739 · Nov 13, 2019
Related Publication 20220398458A1 · Dec 15, 2022
References Cited (51)
US 8126244B2 · Barbu et al. · 2012 [cited by applicant]
US 10181009B2 · Yeatman et al. · 2019 [cited by applicant]
US 10304193B1 · Yu et al. · 2019 [cited by applicant]
US 20080281767A1 · Garner · 2008 [cited by examiner]
US 20100183210A1 · Van Uitert · 2010 [cited by examiner]
US 20120327205A1 · Takahashi · 2012 [cited by applicant]
US 20170218455A1 · Steelman · 2017 [cited by examiner]
US 20170285033A1 · Blume · 2017 [cited by examiner]
US 20180046755A1 · Pe'Er et al. · 2018 [cited by applicant]
US 20180225820A1 · Liang · 2018 [cited by examiner]
US 20180253839A1 · Zur · 2018 [cited by examiner]
US 20180258498A1 · Ahlquist et al. · 2018 [cited by applicant]
US 20180268298A1 · Johansen · 2018 [cited by examiner]
US 20220398458A1 · Mohapatra · 2022 [cited by examiner]
US 20230260111A1 · Rodrigues-Diaz · 2023 [cited by examiner]
WO 2021097302A1 · 2021 [cited by applicant]
International Search Report of related application PCT/US2020/060532, mailed Feb. 5, 2021, 2 pages. [cited by applicant]
Written Opinion of related application PCT/US2020/060532, mailed Feb. 5, 2021, 7 pages. [cited by applicant]
Blanes-Vidal et al., “Addressing priority challenges in the detection and assessment of colorectal polyps from capsule endoscopy and colonoscopy in colorectal cancer screening using machine learning.” Acta Oncologica 58… [cited by applicant]
Byrne et al., “Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model” Gut. Jan. 2019;68(1):94-100. [cited by applicant]
Chen et al., “Accurate Classification of Diminutive Colorectal Polyps Using Computer-Aided Analysis” Gastroenterology. Feb. 2018;154(3):568-575. [cited by applicant]
Danaee et al., “A deep learning approach for cancer detection and relevant gene identification.” Pacific symposium on biocomputing 2017. 14 pages. [cited by applicant]
De Perio “Histopathological Cancer Detectionwith Deep Neural Networks” dated Apr. 20, 2019, retrieved Jan. 8, 2023, 23 pages, from Website URL: https://towardsdatascience.com/histopathological-cancer-detection-with-deep… [cited by applicant]
Gessert et al., “Deep transfer learning methods for colon cancer classification in confocal laser microscopy images.” International journal of computer assisted radiology and surgery 14.11 (2019): 1837-1845. 13 pages. [cited by applicant]
Guo et al., “Automated polyp segmentation for colonoscopy images: A method based on convolutional neural networks and ensemble learning” Med Phys. Dec. 2019;46(12):5666-5676. [cited by applicant]
Ito et al., “Endoscopic Diagnostic Support System for cT1b Colorectal Cancer Using Deep Learning” Oncology. 2019;96(1):44-50. [cited by applicant]
Kainz et al., “Segmentation and classification of colon glands with deep convolutional neural networks and total variation regularization” PeerJ. Oct. 3, 2017;5:e3874. 28 pages. [cited by applicant]
Kather et al., “Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study” PLoS Med. Jan. 24, 2019;16(1):e1002730. 22 pages. [cited by applicant]
Komeda et al., “Computer-Aided Diagnosis Based on Convolutional Neural Network System for Colorectal Polyp Classification: Preliminary Experience” Oncology. 2017;93 Suppl 1:30-34. [cited by applicant]
Mahmood et al., “Deep learning with cinematic rendering: fine-tuning deep neural networks using photorealistic medical images” Phys Med Biol. Sep. 13, 2018;63(18):185012. 17 pages. [cited by applicant]
Ribeiro et al., “Exploring Deep Learning and Transfer Learning for Colonic Polyp Classification” Comput Math Methods Med. 2016;2016:6584725. 16 pages. [cited by applicant]
Roth et al., “Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation” IEEE Trans Med Imaging. May 2016;35(5):1170-81. 35 pages. [cited by applicant]
Tajbakhsh et al., “Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?” IEEE Trans Med Imaging. May 2016;35(5):1299-1312. 17 pages. [cited by applicant]
Urban et al., “Deep Learning Localizes and Identifies Polyps in Real Time With 96% Accuracy in Screening Colonoscopy” Gastroenterology. Oct. 2018;155(4):1069-1078.e8. [cited by applicant]
Vinsard et al., “Quality assurance of computer-aided detection and diagnosis in colonoscopy” Gastrointest Endosc. Jul. 2019;90(1):55-63. [cited by applicant]
Wang et al., ““Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy”” Nat Biomed Eng. Oct. 2018;2(10):741-748. [cited by applicant]
Yao et al., “Automated Detection of Non-Informative Frames for Colonoscopy Through a Combination of Deep Learning and Feature Extraction” Annu Int Conf IEEE Eng Med Biol Soc. Jul. 2019;2019:2402-2406. [cited by applicant]
Yu et al., “Integrating Online and Offline Three-Dimensional Deep Learning for Automated Polyp Detection in Colonoscopy Videos” IEEE J Biomed Health Inform. Jan. 2017;21(1):65-75. [cited by applicant]
Zhang et al., “Automatic Detection and Classification of Colorectal Polyps by Transferring Low-Level CNN Features From Nonmedical Domain” IEEE J Biomed Health Inform. Jan. 2017;21(1):41-47. [cited by applicant]
Rex et al., “The American Society for Gastrointestinal Endoscopy PIVI (Preservation and Incorporation of Valuable Endoscopic Innovations) on real-time endoscopic assessment of the histology of diminutive colorectal poly… [cited by applicant]
Kessler et al., “A quantitative assessment of the risks and cost savings of forgoing histologic examination of diminutive polyps.” Endoscopy (2011): 683-691. [cited by applicant]
Gupta et al., “Accuracy of in vivo optical diagnosis of colon polyp histology by narrow-band imaging in predicting colonoscopy surveillance intervals.” Gastrointestinal endoscopy 75.3 (2012): 494-502. [cited by applicant]
Rasatogi et al., “Recognition of surface mucosal and vascular patterns of colon polyps by using narrow-band imaging: interobserver and intraobserver agreement and prediction of polyp histology.” Gastrointestinal endosco… [cited by applicant]
Rex et al., “Narrow-band imaging without optical magnification for histologic analysis of colorectal polyps.” Gastroenterology 136.4 (2009): 1174-1181. [cited by applicant]
Ladabaum et al., “Real-time optical biopsy of colon polyps with narrow band imaging in community practice does not yet meet key thresholds for clinical decisions.” Gastroenterology 144.1 (2013): 81-91. [cited by applicant]
Rastogi et al., “Impact of a computer-based teaching module on characterization of diminutive colon polyps by using narrow-band imaging by non-experts in academic and community practice: a video-based study.” Gastrointe… [cited by applicant]
Gross et al., “Computer-based classification of small colorectal polyps by using narrow-band imaging with optical magnification.” Gastrointestinal endoscopy 74.6 (2011): 1354-1359. [cited by applicant]
Mori et al., “Impact of an automated system for endocytoscopic diagnosis of small colorectal lesions: an international web-based study.” Endoscopy 48.12 (2016): 1110-1118. [cited by applicant]
Kominami et al., “Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy.” Gastrointestinal endoscopy 83.3 (2016): 643-649. [cited by applicant]
Zachariah et al., “Prediction of polyp pathology using convolutional neural networks achieves ‘resect and discard’ thresholds.” The American journal of gastroenterology 115.1 (2020): 138-44. [cited by applicant]
Yang et al., “Application of artificial intelligence in gastroenterology.” World journal of gastroenterology 25.14 (2019): 1666-83. [cited by applicant]