Granted Patent B2

US 12,367,970 · App. 18/338,957 · Granted Jul 22, 2025

Inventors: Jagadish Venkataraman (Menlo Park, CA); Pablo Garcia Kilroy (Menlo Park, CA)

Assignee: Verb Surgical Inc.

G16H40/20A61B34/10G06N20/00G16H40/40

Abstract

Claims (64)

1. A computer-implemented method for determining a total tool-tissue interaction duration of a surgical procedure on a patient, the method comprising the following operations performed by a digital processor:

obtaining an overall procedure duration of the surgical procedure performed by a surgeon on the patient;

receiving operating room (OR) data collected during the surgical procedure, wherein the OR data includes an endoscope video captured by an endoscope during the overall procedure duration of the surgical procedure;

analyzing, by a machine-learning, ML, model the OR data for the overall procedure duration of the surgical procedure, including detecting all tool-tissue interaction events during the overall procedure duration, as a result of which the ML model identifies a plurality of surgical timeout events for the surgical procedure and a plurality of out of body events in which the endoscope is taken out of the body of the patient and inserted back into the body, wherein the ML model identifies each of the plurality of surgical timeout events whenever a surgical tool in the endoscope video has stopped moving for more than a predetermined time period due to the surgeon pausing a surgical task being performed on the patient and there is no surgical tool-tissue interaction;

extracting a plurality of durations of the plurality of surgical timeout events and a plurality of durations of the plurality of out of body events, respectively;

computing the total tool-tissue interaction duration based on subtracting the plurality of durations of the plurality of surgical timeout events including the plurality of durations of the plurality of out body events, from the overall procedure duration; and

for each of the plurality of out of body events, blurring a segment of the endoscope video that corresponds to an out of body event to anonymize the segment.

2. The computer-implemented method of claim 1 , wherein the OR data further comprises:

audio recorded inside the OR during the surgical procedure; and

wherein the digital processor analyzes for the overall procedure duration the endoscope video and the recorded audio to thereby detect all intervals in which the endoscope video becomes static and the recorded audio contains a discussion between the surgeon and another person.

3. The computer-implemented method of claim 1 , wherein the OR data further comprises audio recorded inside the OR during the surgical procedure, a video captured by a wall camera or a ceiling camera inside the OR during the surgical procedure and sensor data collected inside the OR during the surgical procedure, the sensor data comprises one or more of:

pressure sensor data collected from surgical tools involved in the surgical procedure;

pressure sensor data collected from a surgical platform inside the OR; and

pressure sensor data collected from a doorway of the OR, and wherein

the digital processor analyzes for the overall procedure duration i) the endoscope video, ii) the recorded audio, iii) the video captured by the wall camera or the ceiling camera, and iv) the sensor data, to detect the plurality of surgical timeout events.

4. The computer-implemented method of claim 1 , wherein extracting the plurality of durations of the surgical timeout events comprises for each duration and its associated surgical timeout event:

extracting an initial time being when movement of the surgical tool is determined by the ML model to have stopped; and

extracting an end time being when movement the surgical tool is determined by the ML model to have resumed.

5. The computer-implemented method of claim 4 , further comprising collaborating the initial time and the end time of the associated surgical timeout event with pressure sensor data collected from a pressure sensor located at a tip of the surgical tool.

6. The computer-implemented method of claim 5 , wherein

collaborating the initial time and the end time with the pressure sensor data comprises: collaborating the initial time with a first time when the pressure sensor data decreases to a first threshold; and

collaborating the end time with a second time when the pressure sensor data increases to a second threshold greater than the first threshold.

7. The computer-implemented method of claim 4 , wherein the surgical timeout event occurs:

when the surgeon stops interacting with the patient and starts a discussion with another surgeon, a surgical support team, or a resident surgeon;

when the surgeon pauses to make a decision on how to proceed with the surgical procedure based on an on-screen event or a surgical complication; or

when the surgeon pauses to wait for a collaborating surgeon to come into the OR.

8. The computer-implemented method of claim 1 , wherein the ML model is to:

identify the beginning of the OOB event based on a first sequence of video images in the endoscope video; and

identify the end of the OOB event based on a second sequence of video images in the endoscope video.

9. The computer-implemented method of claim 1 , wherein the OOB event coincides with:

cleaning a lens of the endoscope which blocks an endoscopic view;

changing the lens from one scope size to another scope size; or

switching the surgical procedure from a robotic surgical system to a laparoscopic surgical system.

10. The computer-implemented method of claim 1 , further comprising:

providing part of the OR data as input to another machine-learning model in response to which the other ML model detects a non-surgical event by

identifying a pre-surgery patient preparation time prior to the surgical procedure; and

identifying a post-surgery patient assistant time after completion of the surgical procedure,

wherein determining the total tool-tissue interaction duration is further based on subtracting the pre-surgery patient preparation time and the post-surgery patient assistant time from the overall procedure duration.

11. The computer-implemented method of claim 1 , wherein obtaining the overall procedure duration of the surgical procedure includes determining a time when the patient is being wheeled into the OR and a time when the patient is being wheeled out of the OR.

12. A system for determining a total tool-tissue interaction duration of a surgical procedure on a patient, the system comprising:

a processor; and

a memory coupled to the processor, wherein the memory stores instructions that, when executed by the processor, cause the system to:

obtain an overall procedure duration of the surgical procedure performed by a surgeon on the patient;

receive operating room (OR) data collected during the surgical procedure, wherein the OR data includes an endoscope video captured by an endoscope during the overall procedure duration of the surgical procedure;

analyzing, by a machine-learning model the OR data for the overall procedure duration of the surgical procedure, including detecting all tool-tissue interaction events during the overall procedure duration, as a result of which the ML model identifies a plurality of surgical timeout events for the surgical procedure and one or more out of body events in which the endoscope is taken out of the body of the patient and inserted back into the body, wherein the ML model identifies each of the plurality of surgical timeout events whenever a surgical tool in the endoscope video has stopped moving for more than a predetermined time period due to the surgeon pausing a surgical task being performed on the patient and there is no surgical tool-tissue interaction;

extract a plurality of durations of the plurality of surgical timeout events, respectively, and durations of the one or more out of body events, respectively;

compute the total tool-tissue interaction duration based on subtracting the plurality of durations of the plurality of surgical timeout events including the durations of the one or more out of body events, from the overall procedure duration; and

for each of the one or more out of body events, blurring a segment of the endoscope video that corresponds to an out of body event to anonymize the segment.

13. The system of claim 12 , wherein the OR data further comprises:

audio recorded inside the OR during the surgical procedure; and

wherein the processor analyzes for the overall procedure duration the endoscope video and the recorded audio to thereby detect all intervals in which the endoscope video becomes static and the recorded audio contains a discussion between the surgeon and another person.

14. The system of claim 12 , wherein the OR data further comprises audio recorded inside the OR during the surgical procedure, a video captured by a wall camera or a ceiling camera inside the OR during the surgical procedure and sensor data collected inside the OR during the surgical procedure, the sensor data comprises one or more of:

pressure sensor data collected from surgical tools involved in the surgical procedure;

pressure sensor data collected from a surgical platform inside the OR; and

pressure sensor data collected from a doorway of the OR and wherein

the processor analyzes for the overall procedure duration i) the endoscope video, ii) the recorded audio, iii) the video captured by the wall camera or the ceiling camera, and iv) the sensor data, to detect the plurality of surgical timeout events.

15. The system of claim 12 wherein the instructions cause the system to extract the plurality of durations of the surgical timeout events by, for each duration and its associated surgical timeout event:

extracting an initial time being when movement of the surgical tool is determined by the ML model to have stopped; and

extracting an end time being when movement the surgical tool is determined by the ML model to have resumed.

16. The system of claim 15 wherein the instructions cause the system to collaborate the initial time and the end time of the associated surgical timeout event with pressure sensor data collected from a pressure sensor located at a tip of the surgical tool.

17. The system of claim 15 wherein the surgical timeout event occurs:

when the surgeon stops interacting with the patient and starts a discussion with another surgeon, a surgical support team, or a resident surgeon;

when the surgeon pauses to make a decision on how to proceed with the surgical procedure based on an on-screen event or a surgical complication; or

when the surgeon pauses to wait for a collaborating surgeon to come into the OR.

Assignments (1)

Continuity (3)

References Cited (27)