System and methods for aggregating features in video frames to improve accuracy of AI detection algorithms
Methods and systems are provided for aggregating features in multiple video frames to enhance tissue abnormality detection algorithms, wherein a first detection algorithm identifies an abnormality and aggregates adjacent video frames to create a more complete image for analysis by an artificial intelligence detection algorithm, the aggregation occurring in real time as the medical procedure is being performed.
1. A system for identifying tissue abnormalities in video data generated by an optical endoscopy machine, the endoscopy machine outputting real-time images of an interior of an organ as video frames, the system comprising:
at least one video monitor operably coupled to the endoscopy machine to display the video frames output by the endoscopy machine;
a memory for storing non-volatile programmed instructions; and
a processor configured to accept the video frames output by the endoscopy machine and to store the video frames in the memory, the processor further configured to execute the non-volatile programmed instructions to:
analyze a first video frame using artificial intelligence to determine if any part of a first tissue abnormality is visible within the first video frame, and if the first video frame is determined to include the first tissue abnormality, analyze adjacent video frames to locate other parts of the first tissue abnormality;
generate a reconstructed image of the first tissue abnormality that spans the first video frame and adjacent video frames in which the other parts of the first tissue abnormality are located;
analyze, using artificial intelligence, the reconstructed image to classify the first tissue abnormality;
analyze the reconstructed image to estimate a degree of completeness of the reconstructed image;
display on the at least one video monitor a bounding box surrounding a portion of the reconstructed image that is visible in a current video frame; and
display on the at least one video monitor the estimate of the degree of completeness of the reconstructed image.
2. The system of claim 1 , wherein the programmed instructions, when executed by the processor, generate the reconstructed image of the first tissue abnormality by aggregating at least one of the following in the first video frame and the adjacent video frames: a boundary of the first tissue abnormality, a color of the first tissue abnormality, and a texture of the first tissue abnormality.
3. The system of claim 1 , wherein the programmed instructions, when executed by the processor, generate and display on the at least one video monitor a textual description of a type of the first tissue abnormality.
4. The system of claim 1 , wherein the programmed instructions, when executed by the processor, provide that if analysis of the adjacent video frames does not locate other parts of the first tissue abnormality, the first video frame is analyzed using artificial intelligence to classify the first tissue abnormality and a bounding box is displayed on the at least one video monitor surrounding the first tissue abnormality.
5. The system of claim 4 , wherein the programmed instructions, when executed by the processor, generate and display on the at least one video monitor a textual description of a type of the first tissue abnormality.
6. The system of claim 1 , wherein the processor further is configured to execute the programmed instructions to:
determine a direction of movement of a camera of the colonoscopy machines to acquire additional video frames for use in generating the reconstructed image; and
display on the at least one video monitor an indicator of the direction of movement.
7. The system of claim 1 , wherein the processor further is configured to execute the programmed instructions to:
if analysis of the adjacent video frames detects a second tissue abnormality different from the first tissue abnormality, analyze the adjacent video frames to locate other parts of the second tissue abnormality.
8. The system of claim 1 , wherein the programmed instructions, when executed by the processor, generate a reconstructed image of the first tissue abnormality by adding adjacent features extracted from the adjacent video frames to features extracted from the first video frame.
9. The system of claim 1 , wherein the programmed instructions that implement the artificial intelligence includes a machine learning capability.
10. A method of identifying tissue abnormalities in video data generated by an optical endoscopy machine, the endoscopy machine outputting real-time images of an interior of an organ as video frames, the method comprising:
acquiring the video frames output by the endoscopy machine;
analyzing a first video frame using artificial intelligence to determine if any part of a first tissue abnormality is visible within the first video frame, and if the first video frame is determined to include the first tissue abnormality, analyzing adjacent video frames to locate other parts of the first tissue abnormality;
generating a reconstructed image of the first tissue abnormality that spans the first video frame and adjacent video frames in which the other parts of the first tissue abnormality are located;
determining a direction of movement of a camera of the colonoscopy machines to acquire additional video frames for use in generating the reconstructed image;
analyzing, using artificial intelligence, the reconstructed image to classify the first tissue abnormality;
displaying on at least one video monitor the real time images from the endoscopy machine and a bounding box surrounding a portion of the reconstructed image that is visible in a current video frame; and
displaying on the at least one video monitor an indicator of the direction of movement.
11. The method of claim 10 , wherein generating the reconstructed image of the first tissue abnormality comprises aggregating at least one of the following in the first video frame and the adjacent video frames: a boundary of the first tissue abnormality, a color of the first tissue abnormality, and a texture of the first tissue abnormality.
12. The method of claim 10 , further comprising generating and displaying on the at least one video monitor a textual description of a type of the first tissue abnormality.
13. The method of claim 10 , further comprising, if analysis of the adjacent video frames does not locate other parts of the first tissue abnormality:
analyzing the first video frame using artificial intelligence to classify the first tissue abnormality; and
displaying a bounding box on the at least one video monitor surrounding the first tissue abnormality.
14. The method of claim 13 , further comprising generating and displaying on the at least one video monitor a textual description of a type of the first tissue abnormality.
15. The method of claim 10 , further comprising:
analyzing the reconstructed image to estimate a degree of completeness of the reconstructed image, and
displaying on the at least one video monitor the estimate of the degree of completeness of the reconstructed image.
16. The method of claim 10 , further comprising, if analysis of the adjacent video frames detects a second tissue abnormality different from the first tissue abnormality, analyzing the adjacent video frames to locate other parts of the second tissue abnormality.
17. The method of claim 10 , further comprising generating a reconstructed image of the first tissue abnormality by adding adjacent features extracted from the adjacent video frames to features extracted from the first video frame.
18. The method of claim 10 , further comprising implementing the artificial intelligence to include a machine learning capability.