IP Library › Granted Patent US 11,615,527

Granted Patent B2

US 11,615,527 · App. 16/875,357 · Granted Mar 28, 2023

Automated anatomic and regional location of disease features in colonoscopy videos

Inventors: Kayvan Najarian (Ann Arbor, MI); Heming Yao (Ann Arbor, MI); Sayedmohammadreza Soroushmehr (Ann Arbor, MI); Jonathan Gryak (Ann Arbor, MI); Ryan W. Stidham (Ann Arbor, MI)

Assignee: THE REGENTS OF THE UNIVERSITY OF MICHIGAN

G06T7/0012 G06F16/285 G06K9/6284 G06N3/08 G06T7/269 G06T7/50 G06T7/73 G16H30/20 G06T2207/10016 G06T2207/10024 G06T2207/20081 G06T2207/20084 G06T2207/30028 G06T2207/30244

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,615,527
App. No.: 16/875,357
Granted: Mar 28, 2023
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A system for automatically analyzing a video recording of a colonoscopy includes a processor and memory storing instructions, which when executed by the processor, cause the processor to receive the video recording of the colonoscopy performed on the colon and detect informative frames in the video recording. A frame is informative if the clarity of the frame is above a threshold or if the frame includes clinically relevant information about the colon. The instructions cause the processor to generate scores indicating severity levels of a disease for a plurality of the informative frames, estimate locations of the plurality of the informative frames in the colon, and generate an output indicating a distribution of the scores over one or more segments of the colon by combining the scores generated for the plurality of the informative frames and the estimated locations of the plurality of the informative frames in the colon.

Claims (224)

1. A system for automatically analyzing a video recording of a colonoscopy performed on a colon, the system comprising:

a processor; and

memory storing instructions, which when executed by the processor, cause the processor to:

receive the video recording of the colonoscopy performed on the colon;

detect informative frames in the video recording, wherein a frame is informative if the clarity of the frame is above a threshold or if the frame includes clinically relevant information about the colon;

generate scores indicating severity levels of a disease for a plurality of the informative frames;

estimate locations of the plurality of the informative frames in the colon; and

generate an output indicating a distribution of the scores over one or more segments of the colon by combining the scores generated for the plurality of the informative frames and the estimated locations of the plurality of the informative frames in the colon.

2. The system of claim 1 wherein the instructions cause the processor to generate a summary score based on the scores generated for the plurality of the informative frames and wherein the summary score indicates an overall disease severity for the colon.

3. The system of claim 1 wherein the instructions cause the processor to:

input frames of the video recording to a frame classification model to identify non-informative frames and to distinguish the non-informative frames from the informative frames; and

receive the informative frames from the frames in the video recording from the frame classification model.

4. The system of claim 1 wherein the instructions cause the processor to:

extract features from frames of the video recording used for classifying frames as informative versus non-informative in Red-Green-Blue color space;

input the features to a convolutional neural network;

receive, from the convolutional neural network, bottleneck features from the frames of the video recording in Red-Green-Blue color space;

convert the frames of the video recording into Hue-Saturation-Value color space;

extract other features from the converted frames to identify non-informative frames and to distinguish the non-informative frames from the informative frames; and

generate a frame classification model using a combination of the bottleneck features and the other features to identify the non-informative frames and to distinguish the non-informative frames from the informative frames.

5. The system of claim 1 wherein the instructions cause the processor to:

input the plurality of the informative frames to a convolutional neural network to receive the scores for the plurality of the informative frames; and

receive the scores for the plurality of the informative frames from the convolutional neural network.

6. The system of claim 1 wherein the instructions cause the processor to:

generate a relational database to store metadata including disease findings and scores indicating severity levels of diseases from frames of a plurality of video recordings of colonoscopies; and

generate a convolutional neural network using the metadata and the frames to output the scores for the plurality of the informative frames.

7. The system of claim 1 wherein the instructions cause the processor to:

remove noise from the scores generated for the plurality of the informative frames using a temporal smoothing filter, wherein the scores include a sequence of numbers;

analyze the distribution of the scores;

obtain, based on the analysis, features including a number of sub-sequences of length greater than one in the sequence of numbers per score class;

input the features to a machine learning model; and

receive, from the machine learning model, a summary score indicating an overall disease severity for the colon.

8. The system of claim 1 wherein the instructions cause the processor to estimate the locations of the plurality of the informative frames in the colon by:

estimating motion of a camera used to capture the video recording based on pairs of frames from the plurality of the informative frames and optical flow from the pairs of frames;

generating a template representing relative lengths of segments of the colon;

determining, using the template, relative positions of the plurality of the informative frames in the segments of the colon based on the estimated motion of the camera; and

reordering the plurality of the informative frames according to the relative positions of the plurality of the informative frames in the segments of the colon.

9. The system of claim 8 wherein the instructions cause the processor to refine the estimates of the locations of the plurality of the informative frames in the colon by:

identifying frames from the plurality of the informative frames including distinctive features belonging to selected ones of the segments of the colon; and

identifying frames from the plurality of the informative frames including distinctive features belonging to flexures of the colon.

10. The system of claim 1 wherein the instructions cause the processor to:

generate a deep learning network to estimate a pose of a camera used to capture the video recording, wherein the deep learning network includes a depth network to predict a depth map of the colon visible to the camera and a pose network to predict the pose of the camera;

input a first frame taken at a first time to the depth network;

input a second frame taken at a second time and an optical flow map between the first and second frames to the pose network, wherein the second time is successive to the first time; and

determine the pose of the camera at the second time based on the depth map and the pose predicted respectively by the depth and pose networks, wherein the pose of the camera at the second time indicates a motion of the camera from the first time to the second time.

11. The system of claim 10 wherein the instructions cause the processor to update parameters of the depth and pose networks based on a warping-based image synthesis of the first frame.

12. The system of claim 1 wherein the instructions cause the processor to:

input two consecutive frames from the plurality of the informative frames to a deep learning network to estimate a pose of a camera used to capture the video recording, wherein the deep learning network includes a depth network to predict a depth map of the colon visible to the camera and a pose network to predict the pose of the camera;

receive coordinates of the camera by successively estimating the pose of the camera from the deep learning network;

convert, based on the coordinates, temporal indices of the two consecutive frames into distance indices; and

map, based on the distance indices, the two consecutive frames to one or more segments of the colon.

13. The system of claim 1 wherein the instructions cause the processor to:

input the plurality of the informative frames to a first model to detect frames including distinctive features belonging to selected segments of the colon;

receive, from the first model, frames from the plurality of the informative frames including the distinctive features belonging to the selected segments of the colon;

divide the video recording into a plurality of frame sequences;

detect features of the colon in the frame sequences;

generate a second model to determine, based on the detected features, whether one or more of the frame sequences belong to a flexure of the colon;

input the frame sequences to the second model; and

receive, from the second model, frames from the plurality of the informative frames belonging to a flexure of the colon.

14. A system for automatically analyzing a video recording of a colonoscopy performed on a colon, the system comprising:

a processor; and

memory storing instructions, which when executed by the processor, cause the processor to:

receive the video recording of the colonoscopy performed on the colon;

extract features from frames of the video recording used for classifying frames as informative versus non-informative in Red-Green-Blue color space;

input the features to a convolutional neural network;

receive, from the convolutional neural network, bottleneck features from the frames of the video recording in Red-Green-Blue color space;

convert the frames of the video recording into Hue-Saturation-Value color space;

extract other features from the converted frames to identify non-informative frames and to distinguish the non-informative frames from the informative frames, wherein a frame is informative if the clarity of the frame is above a threshold or if the frame includes clinically relevant information about the colon; and

generate a frame classification model using a combination of the bottleneck features and the other features to automatically identify the non-informative frames and output the informative frames.

15. The system of claim 14 wherein the instructions cause the processor to:

generate scores indicating severity levels of a disease for a plurality of the informative frames;

estimate locations of the plurality of the informative frames in the colon; and

16. The system of claim 15 wherein the instructions cause the processor to generate a summary score based on the scores generated for the plurality of the informative frames and wherein the summary score indicates an overall disease severity for the colon.

17. The system of claim 15 wherein the instructions cause the processor to:

input the plurality of the informative frames to a convolutional neural network to receive the scores for the plurality of the informative frames; and

receive the scores for the plurality of the informative frames from the convolutional neural network.

18. The system of claim 15 wherein the instructions cause the processor to:

generate a relational database to store metadata including disease findings and scores indicating severity levels of diseases from frames of a plurality of video recordings of colonoscopies; and

generate a convolutional neural network using the metadata and the frames to output the scores for the plurality of the informative frames.

19. The system of claim 15 wherein the instructions cause the processor to:

remove noise from the scores generated for the plurality of the informative frames using a temporal smoothing filter, wherein the scores include a sequence of numbers;

analyze the distribution of the scores;

obtain, based on the analysis, features including a number of sub-sequences of length greater than one in the sequence of numbers per score class;

input the features to a machine learning model; and

receive, from the machine learning model, a summary score indicating an overall disease severity for the colon.

20. The system of claim 15 wherein the instructions cause the processor to estimate the locations of the plurality of the informative frames in the colon by:

estimating motion of a camera used to capture the video recording based on pairs of frames from the plurality of the informative frames and optical flow from the pairs of frames;

generating a template representing relative lengths of segments of the colon;

determining, using the template, relative positions of the plurality of the informative frames in the segments of the colon based on the estimated motion of the camera; and

reordering the plurality of the informative frames according to the relative positions of the plurality of the informative frames in the segments of the colon.

21. The system of claim 20 wherein the instructions cause the processor to refine the estimates of the locations of the informative frames in the colon by:

identifying frames from the plurality of the informative frames including distinctive features belonging to selected ones of the segments of the colon; and

identifying frames from the plurality of the informative frames including distinctive features belonging to flexures of the colon.

22. The system of claim 14 wherein the instructions cause the processor to:

input a first frame taken at a first time to the depth network;

input a second frame taken at a second time and an optical flow map between the first and second frames to the pose network, wherein the second time is successive to the first time; and

23. The system of claim 22 wherein the instructions cause the processor to update parameters of the depth and pose networks based on a warping-based image synthesis of the first frame.

24. The system of claim 14 wherein the instructions cause the processor to:

receive coordinates of the camera by successively estimating the pose of the camera from the deep learning network;

convert, based on the coordinates, temporal indices of the two consecutive frames into distance indices; and

map, based on the distance indices, the two consecutive frames to one or more segments of the colon.

25. The system of claim 14 wherein the instructions cause the processor to:

input the plurality of the informative frames to a first model to detect frames including distinctive features belonging to selected segments of the colon;

receive, from the first model, frames from the plurality of the informative frames including the distinctive features belonging to the selected segments of the colon;

divide the video recording into a plurality of frame sequences;

detect features of the colon in the frame sequences;

generate a second model to determine, based on the detected features, whether one or more of the frame sequences belong to a flexure of the colon;

input the frame sequences to the second model; and

receive, from the second model, frames from the plurality of the informative frames belonging to a flexure of the colon.

26. A system for automatically analyzing a video recording of a colonoscopy performed on a colon, the system comprising:

a processor; and

memory storing instructions, which when executed by the processor, cause the processor to:

receive the video recording of the colonoscopy performed on the colon;

input the plurality of the informative frames to a convolutional neural network; and

receive scores indicating severity levels of a disease for the plurality of the informative frames from the convolutional neural network.

27. The system of claim 26 wherein the instructions cause the processor to:

generate a relational database to store metadata including disease findings and scores indicating severity levels of diseases from frames of a plurality of video recordings of colonoscopies; and

generate a convolutional neural network using the metadata and the frames to output the scores for the plurality of the informative frames.

28. The system of claim 26 wherein the instructions cause the processor to generate a summary score based on the scores generated for the plurality of the informative frames and wherein the summary score indicates an overall disease severity for the colon.

29. The system of claim 26 wherein the instructions cause the processor to:

estimate locations of the plurality of the informative frames in the colon; and

30. The system of claim 26 wherein the instructions cause the processor to:

input frames of the video recording to a frame classification model to identify non-informative frames and to distinguish the non-informative frames from the informative frames; and

receive the informative frames from the frames in the video recording from the frame classification model.

31. The system of claim 26 wherein the instructions cause the processor to:

extract features from frames of the video recording used for classifying frames as informative versus non-informative in Red-Green-Blue color space;

input the features to a convolutional neural network;

receive, from the convolutional neural network, bottleneck features from the frames of the video recording in Red-Green-Blue color space;

convert the frames of the video recording into Hue-Saturation-Value color space;

extract other features from the converted frames to identify non-informative frames and to distinguish the non-informative frames from the informative frames; and

32. The system of claim 29 wherein the instructions cause the processor to:

input the plurality of the informative frames to a convolutional neural network to receive the scores for the plurality of the informative frames; and

receive the scores for the plurality of the informative frames from the convolutional network.

33. The system of claim 29 wherein the instructions cause the processor to estimate the locations of the plurality of the informative frames in the colon by:

estimating motion of a camera used to capture the video recording based on pairs of frames from the plurality of the informative frames and optical flow from the pairs of frames;

generating a template representing relative lengths of segments of the colon;

determining, using the template, relative positions of the plurality of the informative frames in the segments of the colon based on the estimated motion of the camera; and

reordering the plurality of the informative frames according to the relative positions of the plurality of the informative frames in the segments of the colon.

34. The system of claim 33 wherein the instructions cause the processor to refine the estimates of the locations of the plurality of the informative frames in the colon by:

identifying frames from the plurality of the informative frames including distinctive features belonging to selected ones of the segments of the colon; and

identifying frames from the plurality of the informative frames including distinctive features belonging to flexures of the colon.

35. The system of claim 26 wherein the instructions cause the processor to:

input a first frame taken at a first time to the depth network;

input a second frame taken at a second time and an optical flow map between the first and second frames to the pose network, wherein the second time is successive to the first time; and

36. The system of claim 35 wherein the instructions cause the processor to update parameters of the depth and pose networks based on a warping-based image synthesis of the first frame.

37. The system of claim 26 wherein the instructions cause the processor to:

receive coordinates of the camera by successively estimating the pose of the camera from the deep learning network;

convert, based on the coordinates, temporal indices of the two consecutive frames into distance indices; and

map, based on the distance indices, the two consecutive frames to one or more segments of the colon.

38. The system of claim 26 wherein the instructions cause the processor to:

input the plurality of the informative frames to a first model to detect frames including distinctive features belonging to selected segments of the colon;

receive, from the first model, frames from the plurality of the informative frames including the distinctive features belonging to the selected segments of the colon;

divide the video recording into a plurality of frame sequences;

detect features of the colon in the frame sequences;

generate a second model to determine, based on the detected features, whether one or more of the frame sequences belong to a flexure of the colon;

input the frame sequences to the second model; and

receive, from the second model, frames from the plurality of the informative frames belonging to a flexure of the colon.

39. A system for automatically analyzing a video recording of a colonoscopy performed on a colon, the system comprising:

a processor; and

memory storing instructions, which when executed by the processor, cause the processor to:

receive the video recording of the colonoscopy performed on the colon;

estimate locations of a plurality of the informative frames in the colon by:

estimating motion of a camera used to capture the video recording based on pairs of frames from the plurality of the informative frames and optical flow from the pairs of frames;

generating a template representing relative lengths of segments of the colon;

determining, using the template, relative positions of the plurality of the informative frames in the segments of the colon based on the estimated motion of the camera; and

reordering the plurality of the informative frames according to the relative positions of the plurality of the informative frames in the segments of the colon.

40. The system of claim 39 wherein the instructions cause the processor to:

generate scores indicating severity levels of a disease for the plurality of the informative frames; and

41. The system of claim 40 wherein the instructions cause the processor to generate a summary score based on the scores generated for the plurality of the informative frames and wherein the summary score indicates an overall disease severity for the colon.

42. The system of claim 39 wherein the instructions cause the processor to:

input frames of the video recording to a frame classification model to identify non-informative frames and to distinguish the non-informative frames from the informative frames; and

receive the informative frames from the frames in the video recording from the frame classification model.

43. The system of claim 39 wherein the instructions cause the processor to:

extract features from frames of the video recording used for classifying frames as informative versus non-informative in Red-Green-Blue color space;

input the features to a convolutional neural network;

receive, from the convolutional neural network, bottleneck features from the frames of the video recording in Red-Green-Blue color space;

convert the frames of the video recording into Hue-Saturation-Value color space;

extract other features from the converted frames to identify non-informative frames and to distinguish the non-informative frames from the informative frames; and

44. The system of claim 40 wherein the instructions cause the processor to:

input the plurality of the informative frames to a convolutional neural network to receive the scores for the plurality of the informative frames; and

receive the scores for the plurality of the informative frames from the convolutional neural network.

45. The system of claim 39 wherein the instructions cause the processor to:

generate a relational database to store metadata including disease findings and scores indicating severity levels of diseases from frames of a plurality of video recordings of colonoscopies; and

generate a convolutional neural network using the metadata and the frames to output scores for the plurality of the informative frames.

46. The system of claim 40 wherein the instructions cause the processor to:

remove noise from the scores generated for the plurality of the informative frames using a temporal smoothing filter, wherein the scores include a sequence of numbers;

analyze the distribution of the scores;

obtain, based on the analysis, features including a number of sub-sequences of length greater than one in the sequence of numbers per score class;

input the features to a machine learning model; and

receive, from the machine learning model, a summary score indicating an overall disease severity for the colon.

47. The system of claim 39 wherein the instructions cause the processor to refine the estimates of the locations of the informative frames in the colon by:

identifying frames from the plurality of the informative frames including distinctive features belonging to selected ones of the segments of the colon; and

identifying frames from the plurality of the informative frames including distinctive features belonging to flexures of the colon.

48. The system of claim 39 wherein the instructions cause the processor to:

input a first frame taken at a first time to the depth network;

input a second frame taken at a second time and an optical flow map between the first and second frames to the pose network, wherein the second time is successive to the first time; and

49. The system of claim 48 wherein the instructions cause the processor to update parameters of the depth and pose networks based on a warping-based image synthesis of the first frame.

50. The system of claim 39 wherein the instructions cause the processor to:

receive coordinates of the camera by successively estimating the pose of the camera from the deep learning network;

convert, based on the coordinates, temporal indices of the two consecutive frames into distance indices; and

map, based on the distance indices, the two consecutive frames to one or more segments of the colon.

51. The system of claim 39 wherein the instructions cause the processor to:

input the plurality of the informative frames to a first model to detect frames including distinctive features belonging to selected segments of the colon;

receive, from the first model, frames from the plurality of the informative frames including the distinctive features belonging to the selected segments of the colon;

divide the video recording into a plurality of frame sequences;

detect features of the colon in the frame sequences;

generate a second model to determine, based on the detected features, whether one or more of the frame sequences belong to a flexure of the colon;

input the frame sequences to the second model; and

receive, from the second model, frames from the plurality of the informative frames belonging to a flexure of the colon.

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 11, 2020

From: NAJARIAN, KAYVAN; YAO, HEMING; SOROUSHMEHR, SAYEDMOHAMMADREZA; GRYAK, JONATHAN; STIDHAM, RYAN W.

To: THE REGENTS OF THE UNIVERSITY OF MICHIGAN

Reel/Frame 054335/0470 →

Continuity (2)

Provisional Application 62848724 · May 16, 2019

Related Publication 20200364859A1 · Nov 19, 2020

Cited By (1)

US 12,727,739