Systems and methods for processing real-time video from a medical image device and detecting objects in the video
The present disclosure relates to systems and methods for processing real-time video and detecting objects in the video. In one implementation, a system is provided that includes an input port for receiving real-time video obtained from a medical image device, a first bus for transferring the received real-time video, and at least one processor configured to receive the real-time video from the first bus, perform object detection by applying a trained neural network on frames of the received real-time video, and overlay a border indicating a location of at least one detected object in the frames. The system also includes a second bus for receiving the video with the overlaid border, an output port for outputting the video with the overlaid border from the second bus to an external display, and a third bus for directly transmitting the received real-time video to the output port.
1. A computer-implemented system for real-time video processing, comprising:
a medical imaging device configured to capture real-time video during a medical procedure, the real-time video comprising a plurality of frames;
one or more neural networks that implement:
a trained object detector configured to directly receive the plurality of frames captured by the medical imaging device as input and to generate at least one detection of a feature-of-interest in the plurality of frames as output; and
a trained classifier configured to receive the at least one detection of the feature-of-interest from the trained object detector as input and to generate, based on at least one category, a classification of the feature-of-interest as output;
a display device configured to display to an operator of the medical imaging device, in real-time, the plurality of frames together with an overlay based on the at least one detection of the feature-of-interest and the classification of the feature-of-interest;
an input device configured to receive from the operator a command that adjusts a sensitivity setting associated with at least one of the detection and the classification; and
at least one processor configured to adjust, during the medical procedure in response to the command, one or more weights of one or more nodes of at least one of the trained object detector and the trained classifier.
2. The system of claim 1 , wherein the medical procedure is at least one of an endoscopy, a gastroscopy, a colonoscopy, and an enteroscopy.
3. The system of claim 1 , wherein the feature-of-interest includes an abnormality of human tissue comprising at least one of a formation on or of the human tissue, a change in the human tissue from one type of cell to another type of cell, an absence of the human tissue from a location where the human tissue is expected, and a lesion.
4. The system of claim 1 , wherein the at least one processor is further configured to adjust, in response to the command, one or more thresholds associated with an output layer of at least one of the trained object detector and the trained classifier.
5. The system of claim 1 , wherein the overlay indicates a location of the at least one detection of the feature-of-interest in the plurality of frames.
6. The system of claim 5 , wherein the overlay comprises a bounding box.
7. The system of claim 5 , wherein the display of the overlay is modified based on the classification generated by the trained classifier.
8. The system of claim 7 , wherein the display of the overlay is modified to a first color if the trained classifier classifies the feature-of-interest in a first category, and is modified to a second color if the trained classifier classifies the feature-of-interest in a second category.
9. The system of claim 1 , wherein the classification is based on at least one of a histological classification, morphological classification, and a structural classification.
10. The system of claim 1 , wherein one or more weights of one or more nodes of the trained object detector is adjusted based on the sensitivity setting; and wherein a number of detections that are produced by the trained object detector is increased or decreased in accordance with the sensitivity setting.
11. A method for real-time video processing, the method comprising:
capturing, via a medical imaging device, real-time video during a medical procedure, the real-time video comprising a plurality of frames;
directly providing the real-time video comprising the plurality of frames to a trained object detector, the trained object detector comprising one or more neural networks;
generating at least one detection of a feature-of-interest by applying the trained object detector to the plurality of frames;
generating, based on at least one category, a classification of the feature-of-interest by applying a trained classifier to the at least one detection, the trained classifier comprising one or more neural networks;
displaying to an operator of the medical imaging device, via a display device, the plurality of frames together with an overlay based on the at least one detection of the feature-of-interest and the classification of the feature-of-interest;
receiving from the operator, via an input device, a command that adjusts a sensitivity setting associated with at least one of the detection and the classification; and
adjusting, during the medical procedure in response to the command, one or more weights of one or more nodes of at least one of the trained object detector and the trained classifier.
12. The method of claim 11 , wherein the medical procedure is at least one of an endoscopy, a gastroscopy, a colonoscopy, and an enteroscopy.
13. The method of claim 11 , wherein the feature-of-interest includes an abnormality of human tissue comprising at least one of a formation on or of the human tissue, a change in the human tissue from one type of cell to another type of cell, an absence of the human tissue from a location where the human tissue is expected, and a lesion.
14. The method of claim 11 , further comprising:
adjusting, in response to the command, one or more thresholds associated with an output layer of at least one of the trained object detector and the trained classifier.
15. The method of claim 11 , wherein the overlay indicates a location of the at least one detection of the feature-of-interest in the plurality of frames.
16. The method of claim 15 , wherein the overlay comprises a bounding box.
17. The method of claim 15 , further comprising:
modifying the display of the overlay based on the classification generated by the trained classifier.
18. The method of claim 17 , further comprising:
modifying the display of the overlay to a first color if the trained classifier classifies the feature-of-interest in a first category; and
modifying the overlay to a second color if the trained classifier classifies the feature-of-interest in a second category.
19. The method of claim 11 , wherein the classification is based on at least one of a histological classification, morphological classification, and a structural classification.
20. The method of claim 11 , wherein one or more weights of one or more nodes of the trained object detector is adjusted based on the sensitivity setting; and wherein a number of detections that are produced by the trained object detector is increased or decreased in accordance with the sensitivity setting.