Method for real-time detection of objects, structures or patterns in a video, an associated system and an associated computer readable medium
This invention relates to methods and systems for the real-time detection of objects, structures and/or patterns in videos, such as anatomical structures and/or anatomical landmarks in endoscopic videos of a subject, e.g. endoscopic videos of the gastrointestinal tract (GI tract).
1. A computer-implemented method for real-time detection of polyps in a video from a colonoscopy, the method comprising
a) receiving a sequence of frames of the video from the colonoscopy;
b) applying a sliding window to the sequence of frames over time, and for each time position of the sliding window extracting one or more visual features from a plurality of frames within the sliding window to generate a respective time image, thereby generating a plurality of time images;
c) applying a trained classifier to each time image, wherein the trained classifier determines one or more detection scores that indicate likelihoods that the respective time image contains one or more polyps; and
d) outputting in real-time a detection of said one or more polyps when the one or more detection scores are higher than a detection threshold of the trained classifier.
2. The method of claim 1 , wherein the size of the sliding window is fixed or dynamic.
3. The method of claim 1 , wherein the sliding window is overlapping or non-overlapping.
4. The method of claim 1 , wherein a sliding rate of the sliding window and a frame rate of the video from the colonoscopy are identical.
5. The method of claim 1 , wherein the one or more visual features are extracted by employing algorithms for local feature extraction and/or global feature extraction and/or deep feature extraction.
6. The method of claim 1 , wherein the one or more visual features are deep features and said deep features are extracted through deep neural networks (DNN).
7. The method of claim 1 , wherein the classifier is trained for multi-class classification.
8. The method of claim 1 , wherein the output is in the form of a detection signal selected from the group consisting of-a visual alert and an audio alert and/or is in the form of data stored on a data storage unit.
9. The method of claim 1 , further comprising one or more pre-processing steps prior to step b), wherein said pre-processing steps are selected from the group consisting of noise removal, removal of black borders, cropping, resizing, blurring of edges, and removal of metadata.
10. A system for real-time detection of polyps in a video from a colonoscopy, wherein said system comprises (i) an input configured to receive a sequence of frames of the video from the colonoscopy, (ii) a processing system configured to access and process the sequence of frames and (iii) an output configured to output in real-time a detection of said polyps, wherein the processing system is configured to process the sequence of frames with the steps comprising:
a) receiving the sequence of frames of the video from the colonoscopy;
b) applying a sliding window to the sequence of frames over time, and for each time position of the sliding window extracting one or more visual features from said a plurality of frames within the sliding window to generate a respective time image, thereby generating a plurality of time images;
c) applying a trained classifier to each time image, wherein the trained classifier determines one or more detection scores that indicate likelihoods that the respective time image contains one or more polyps; and
d) outputting in real-time the detection of said one or more polyps when the one or more detection scores are higher than a detection threshold of the trained classifier.
11. The system of claim 10 , wherein the size of the sliding window is fixed or dynamic.
12. The system of claim 10 , wherein the sliding window is overlapping or non-overlapping.
13. The system of claim 10 , wherein a sliding rate of the sliding window and a frame rate of the input video are identical.
14. The system of claim 10 , wherein the one or more visual features are extracted by employing algorithms for local feature extraction and/or global feature extraction and/or deep feature extraction.
15. The system of claim 10 , wherein the visual features are deep features and said deep features are extracted through deep neural networks (DNN).
16. The system of claim 10 , wherein the classifier is trained for multi-class classification.
17. The system of claim 10 , wherein the output is in the form of a detection signal selected from group consisting of a visual alert and an audio alert and/or is in the form of data stored on a data storage unit.
18. The system of claim 17 , wherein the output is in the form of a visual alert which is overlaid over a video feed from the colonoscopy to result in an overlay video which is displayed on a monitor.
19. The system of claim 10 , wherein the steps further comprise one or more pre-processing steps prior to step b), wherein the one or more pre-processing steps are selected from the group consisting of noise removal, removal of black borders, cropping, resizing, blurring of edges, and removal of metadata.
20. The system of claim 10 , wherein the system further comprises graphics hardware comprising a video compositor configured to operate independently of the processing system, ensuring that the video is displayed on a monitor in real-time, even if the system fails.
21. A colonoscope system comprising the system of claim 10 and a colonoscope connected to or in communication with the input of said system.
22. A non-transitory, computer readable medium storing computer readable instructions for real-time detection of polyps in a video from a colonoscopy, wherein the computer program instructions, when executed by a processing system, cause the processing system to perform operations comprising:
a) receiving a sequence of frames of the video from the colonoscopy;
b) applying a sliding window to the sequence of frames over time, and for each time position of the sliding window extracting one or more visual features from a plurality of frames within the sliding window to generate a respective time image, thereby generating a plurality of time images;
c) applying a trained classifier to each time image, wherein the trained classifier determines one or more detection scores that indicate likelihoods that the respective time image contains one or more polyps; and
d) outputting in real-time the detection of said one or more polyps when the one or more detection scores are higher than a detection threshold of the trained classifier.