Real-time surgical tool presence/absence detection in surgical videos
Embodiments described herein provide various techniques and systems for building machine-learning surgical tool presence/absence detection models for processing surgical videos and predicting whether a surgical tool is present or absent in each video frame of a surgical video. In one aspect, a process for ensuring patient safety during a laparoscopic or robotic surgery involving an energy tool is disclosed. The process can begin receiving a real-time control signal indicating an operating state of an energy tool during the surgery. Next, the process receives real-time endoscope video images of the surgery. The process simultaneously applies a machine-learning surgical tool presence/absence detection model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images. The process then checks the real-time control signal against the real-time decisions to identify an unsafe event and takes a proper action when an unsafe event is identified.
1. A computer-implemented method for ensuring patient safety during a laparoscopic or robotic surgery involving an energy tool, the method comprising:
receiving a real-time tool control signal indicating an operating state of an energy tool during the surgery, the real-time tool control signal includes a plurality of activation pulses and wherein each activation pulse in the plurality of activation pulses corresponds to a time duration when the energy tool is activated;
simultaneously receiving real-time endoscope video images of the surgery;
simultaneously applying a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame and a confidence level associated with the tool presence/absence decision;
identifying an unsafe event in real-time by determining whenever the tool presence/absence decision is a tool absence decision that coincides with an activation pulse of the plurality of activation pulses; and
taking a proper action when the unsafe event is identified.
2. The computer-implemented method of claim 1 , wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and wherein the machine-learning model is trained to generate a tool presence decision for a video frame only when both of the two jaws are detected in the video frame.
3. The computer-implemented method of claim 1 wherein identifying the unsafe event further comprises
determining if the tool absence decision is at the beginning of the time duration of the activation pulse in the real-time tool control signal; and
if so, determining that the unsafe event is identified;
otherwise, determining that the energy tool is safe to use.
4. The computer-implemented method of claim 3 , wherein if the tool absence decision does not coincide with any activation pulse in the real-time tool control signal, the method further comprises determining that the energy tool is safe to use.
5. The computer-implemented method of claim 1 , wherein taking the proper action when the unsafe event is identified includes:
if the confidence level is above a high confidence level threshold, immediately disabling the energy tool; and
if the confidence level is below the high confidence level threshold, taking one or more actions selected from the following options without disabling the energy tool:
displaying a visual alert on an endoscope monitor;
generating an audio alert;
generating a mechanical vibration through the energy tool; and
delaying the firing of the energy tool until a user takes a further action on the energy tool.
6. A system for ensuring patient safety during a laparoscopic or robotic surgery involving an energy tool, the system comprising:
one or more processors; and
a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the system to:
receive a real-time tool control signal indicating an operating state of an energy tool during the surgery, the real-time tool control signal includes a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated;
simultaneously receive real-time endoscope video images of the surgery;
simultaneously apply a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame and a confidence level associated with the tool presence/absence decision;
identifying an unsafe event by determining when the tool presence/absence decision is a tool absence decision that coincides with an activation pulse of the plurality of activation pulses; and
take a proper action when the unsafe event is identified.
7. The system of claim 6 , wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and wherein the machine-learning model is trained to generate a tool presence decision for a video frame only when both of the two jaws are detected in the video frame.
8. The system of claim 6 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to identify the unsafe event by:
determining if the tool absence decision is at the beginning of the time duration of the activation pulse in the real-time tool control signal; and
if so, determining that the unsafe event is identified;
otherwise, determining that the energy tool is safe to use.
9. The system of claim 8 , wherein if the tool absence decision does not coincide with any activation pulse in the real-time tool control signal, the system determines that the energy tool is safe to use.
10. The system of claim 6 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the system to take the proper action when the unsafe event is identified by:
immediately disabling the energy tool if the confidence level is above a high confidence level threshold; and
taking one or more actions selected from the following options without disabling the energy tool if the confidence level is below the high confidence level threshold:
displaying a visual alert on an endoscope monitor;
generating an audio alert;
generating a mechanical vibration through the energy tool; and
delaying the firing of the energy tool until a user takes a further action on the energy tool.
11. An ultrasonic or bipolar tissue cutting/cauterizing surgical system, the surgical system comprising:
a signal and power generator;
an energy tool;
one or more processors; and
a memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the surgical system to:
receive a real-time tool control signal from the generator, wherein the real-time tool control signal indicates an operating state of the energy tool during a surgery, the real-time tool control signal includes a plurality of activation pulses, each activation pulse represents a time duration when the energy tool is activated;
simultaneously receive real-time endoscope video images of the surgery;
simultaneously apply a machine-learning model to the real-time endoscope video images to generate real-time decisions on a location of the energy tool in the real-time endoscope video images wherein for each video frame in the real-time endoscope video images a tool presence/absence decision is generated indicating whether the energy tool is present or absent in the video frame and a confidence level associated with the tool presence/absence decision;
identify an unsafe event by determining when the tool presence/absence decision is a tool absence decision that coincides with an activation pulse of the plurality of activation pulses; and
take a proper action when the unsafe event is identified.
12. The surgical system of claim 11 , wherein the energy tool is an ultrasonic energy tool for cutting and sealing tissues at the same time using two jaws, and wherein the machine-learning model is trained to generate a tool presence decision for a video frame only when both of the two jaws are detected in the video frame.
13. The surgical system of claim 11 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the surgical system to identify the unsafe event by:
determining if the tool absence decision is at the beginning of the time duration of the activation pulse in the real-time tool control signal; and
if so, determining that the unsafe event is identified;
otherwise, determining that the energy tool is safe to use.