Detection and evaluation of a surgical time-out confirming surgical details from captured video of an operating room
Multiple image capture devices are located in an operating room to capture video of the entirety of the operating room. A surgical tracking server applies one or more models to the captured video to determine states of objects in the operating room from which a phase of the operating room is determined. A surgical time-out where information about a surgical procedure is audibly provided by people in the operating room during the surgical procedure. From audio and video of the operating room, the surgical tracking server determines whether the time-out was performed and identifies audio and video corresponding to the surgical time-out. The identified audio and video are compared to criteria to evaluate completeness of the surgical time-out. A quality metric for the time out based on the captured video data based on characteristics of the video or audio of the operating room may also be generated.
1 . A method comprising:
obtaining video of an operating room from a plurality of image capture devices positioned within the operating room, different image capture devices having different locations within the operating room;
obtaining audio of the operating room;
comparing, through application of one or more models by a surgical tracking server, the audio and video of the operating room to one or more criteria corresponding to performance of a surgical time-out; and
in response to determining that the obtained audio and video of the operating room satisfy one or more criteria corresponding to initiating the surgical time-out, storing an indication that the surgical time-out has been initiated with an identifier of the operating room;
generating a quality metric indicating completion of the surgical time-out in response to audio and video data of the operating room collected after the surgical time-out has been initiated satisfying a threshold amount of quality criteria, wherein the quality metric is generated by applying one or more trained models to the audio and video data of the operating room collected after the surgical time-out has been initiated and the quality criteria; and
storing an indication of the completion of the surgical time-out in association with the audio and video of the operating room collected after the surgical time-out has been initiated.
2 . The method of claim 1 , wherein the one or more obtained criteria correspond to initiating the surgical time-out, the one or more obtained criteria comprising a stored word or a stored phrase.
3 . The method of claim 2 , wherein comparing, through application of one or more models, the audio and video of the operating room to the one or more criteria corresponding to performance of the surgical time-out comprises:
determining whether audio of the operating room includes the stored word or the stored phrase.
4 . The method of claim 1 , wherein the quality criteria comprise one or more criteria corresponding to the completion of the surgical time-out, the method further comprising:
comparing audio and video of the operating room collected after the surgical time-out has been initiated to one or more criteria corresponding to the completion of the surgical time-out;
in response to the comparing of the audio and video of the operating room collected after the surgical time-out has been initiated to one or more criteria corresponding to the completion of the surgical time-out, determining the audio and video of the operating room collected after the surgical time-out has been initiated satisfy the one or more criteria corresponding to the completion of the surgical time-out; and
stopping storing the indication of the completion of the surgical time-out.
5 . The method of claim 4 , further comprising:
storing an ending indicator in association with the time when the audio and video of the operating room collected after the surgical time-out has been initiated were determined to satisfy the one or more criteria corresponding to the completion of the surgical time-out.
6 . The method of claim 1 , wherein generating a quality metric indicating completion of the surgical time-out comprises:
generating a completion metric indicating completion of the surgical time-out in response to content from the audio and video of the operating room collected after the surgical time-out has been initiated satisfying at least a threshold amount of quality criteria.
7 . The method of claim 1 , wherein the quality criteria comprises one or more of the following: audible identification of each person in the operating room participating in a surgical procedure and their role in the surgical procedure, audible identification of a patient's identity, audible identification of the surgical procedure to be performed, audible identification of a surgical site on which the surgical procedure is to be performed, audible identification of patient-specific concerns for anesthesia, audible identification of sterility of instruments to be used in the surgical procedure, audible identification of one or more steps in the surgical procedure, and any combination thereof.
8 . The method of claim 6 , wherein the completion metric comprises a score based on an amount of quality criteria satisfied by the audio and video of the operating room collected after the surgical time-out has been initiated.
9 . The method of claim 1 , wherein generating a quality metric indicating completion of the surgical time-out comprises:
determining an ambient noise level in the operating room during the audio and video of the operating room collected after the surgical time-out has been initiated; and
generating a quality metric indicating whether the ambient noise level exceeded a threshold.
10 . The method of claim 1 , wherein generating a quality metric indicating completion of the surgical time-out comprises:
determining a number of people moving in and out of a region of the operating room identified in video of the operating room collected after the surgical time-out has been initiated.
11 . The method of claim 1 , further comprising:
transmitting a notification to a display in the operating room based on the quality metric.
12 . The method of claim 11 , wherein transmitting the notification to the display in the operating room based on the quality metric comprises:
transmitting the notification to the display in the operating room in response to determining that each identified person in the audio and video of the operating room has identified themselves and their role in a surgical procedure.
13 . The method of claim 11 , wherein transmitting the notification to the display in the operating room based on the one or more quality metrics comprises:
transmitting the notification to the display in the operating room in response to determining that at least one person in the audio and video of the operating room collected after the surgical time-out has been initiated has not identified themselves or their role in a surgical procedure.
14 . The method of claim 1 , further comprising:
determining a sub-phase of the operating room from the audio and video of the operating room collected after the surgical time-out has been initiated;
retrieving a specific sub-phase of the operating room during which the surgical time-out is to be performed; and
in response to the determined sub-phase of the operating room being a sub-phase that is later than the specific sub-phase and in response to the indication that the surgical time-out has been initiated not being stored, transmitting a notification to a display included in the operating room including a prompt to perform the surgical time-out.
15 . The method of claim 1 , further comprising:
determining a sub-phase of the operating room from the audio and video of the operating room collected after the surgical time-out has been initiated;
retrieving a specific sub-phase of the operating room during which the surgical time-out is to be performed;
retrieving a predicted duration for the specific sub-phase; and
in response to a length of time the operating room has been in the determined sub-phase of the operating room being within a threshold amount of time from the predicted time and in response to the indication that the surgical time-out has been initiated not being stored, transmitting a notification to a display included in the operating room including a prompt to perform the surgical time-out.
16 . A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:
obtain video of an operating room from a plurality of image capture devices positioned within the operating room, different image capture devices having different locations within the operating room;
obtain audio of the operating room;
compare, through application of one or more models by a surgical tracking server, the audio and video of the operating room to one or more criteria corresponding to performance of a surgical time-out; and
in response to determining that the obtained audio and video of the operating room satisfy one or more criteria corresponding to initiating the surgical time-out, store an indication that the surgical time-out has been initiated with an identifier of the operating room;
generate a quality metric indicating completion of the surgical time-out in response to audio and video data of the operating room collected after the surgical time-out has been initiated satisfying a threshold amount of quality criteria, wherein the quality metric is generated by applying one or more trained models to the audio and video data of the operating room collected after the surgical time-out has been initiated and the quality criteria; and
store an indication of the surgical time-out in association with subsequently received audio and video of the operating room obtained after a time when the audio and video of the operating room were determined to satisfy the one or more criteria corresponding to initiating the surgical time-out.
17 . The computer program product of claim 16 , wherein the one or more obtained criteria correspond to initiating the surgical time-out, the one or more obtained criteria comprising a stored word or a stored phrase.
18 . The computer program product of claim 17 , wherein compare, through application of one or more models, the audio and video of the operating room to the one or more criteria corresponding to performance of the surgical time-out comprises:
determine whether audio of the operating room includes the stored word or the stored phrase.
19 . The computer program product of claim 16 , wherein the quality criteria comprise one or more criteria corresponding to the completion of the surgical time-out, wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, further cause the processor to:
compare audio and video of the operating room collected after the surgical time-out has been initiated to one or more criteria corresponding to the completion of the surgical time-out;
in response to the comparing of the audio and video of the operating room collected after the surgical time-out has been initiated to one or more criteria corresponding to the completion of the surgical time-out, determine the audio and video of the operating room collected after the surgical time-out has been initiated satisfy the one or more criteria corresponding to the completion of the surgical time-out; and
stop storing the indication of the completion of the surgical time-out.
20 . The computer program product of claim 19 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:
store an ending indicator in association with the time when the audio and video of the operating room collected after the surgical time-out has been initiated were determined to satisfy the one or more criteria corresponding to the completion of the surgical time-out.
21 . The computer program product of claim 16 , wherein instructions to generate a quality metric indicating completion of the surgical time-out further comprises instructions that cause the processor to:
generate a completion metric indicating completion of the surgical time-out in response to content from the audio and video of the operating room collected after the surgical time-out has been initiated satisfying at least a threshold amount of quality criteria.
22 . The computer program product of claim 21 , wherein the completion metric comprises a score based on an amount of quality criteria satisfied by the audio and video of the operating room collected after the surgical time-out has been initiated.
23 . The computer program product of claim 16 , wherein the quality criteria comprises one or more of the following: audible identification of each person in the operating room participating in a surgical procedure and their role in the surgical procedure, audible identification of a patient's identity, audible identification of the surgical procedure to be performed, audible identification of a surgical site on which the surgical procedure is to be performed, audible identification of patient-specific concerns for anesthesia, audible identification of sterility of instruments to be used in the surgical procedure, audible identification of one or more steps in the surgical procedure, and any combination thereof.
24 . The computer program product of claim 16 , wherein instructions to generate a quality metric indicating completion of the surgical time-out further comprises instructions that cause the processor to:
determine an ambient noise level in the operating room during the audio and video of the operating room collected after the surgical time-out has been initiated; and
generate a quality metric indicating whether the ambient noise level exceeded a threshold.
25 . The computer program product of claim 16 , wherein instructions to generate a quality metric indicating completion of the surgical time-out further comprises instructions that cause the processor to:
determine a number of people moving in and out of a region of the operating room identified in video of the operating room collected after the surgical time-out has been initiated.
26 . The computer program product of claim 16 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to:
transmit a notification to a display in the operating room based on the quality metric.
27 . The computer program product of claim 26 , wherein instructions to transmit the notification to the display in the operating room based on the one or more quality metrics further comprises instructions that cause the processor to:
transmit the notification to the display in the operating room in response to determining that each identified person in the audio and video of the operating room has identified themselves and their role in a surgical procedure.
28 . The computer program product of claim 26 , wherein instructions to transmit the notification to the display in the operating room based on the one or more quality metrics further comprises instructions that cause the processor to:
transmit the notification to the display in the operating room in response to determining that at least one person in the audio and video of the operating room collected after the surgical time-out has been initiated has not identified themselves or their role in a surgical procedure.
29 . The computer program product of claim 16 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon, that when executed by the processor, cause the processor to:
determine a sub-phase of the operating room from the audio and video of the operating room collected after the surgical time-out has been initiated;
retrieve a specific sub-phase of the operating room during which the surgical time-out is to be performed; and
in response to the determined sub-phase of the operating room being a sub-phase that is later than the specific sub-phase and in response to the indication that the surgical time-out has been initiated not being stored, transmit a notification to a display included in the operating room including a prompt to perform the surgical time-out.
30 . The computer program product of claim 16 , wherein the non-transitory computer readable storage medium further has instructions encoded thereon, that when executed by the processor, cause the processor to:
determine a sub-phase of the operating room from the audio and video of the operating room collected after the surgical time-out has been initiated;
retrieve a specific sub-phase of the operating room during which the surgical time-out is to be performed;
retrieve a predicted duration for the specific sub-phase; and
in response to a length of time the operating room has been in the determined sub-phase of the operating room being within a threshold amount of time from the predicted time and in response to the indication that the surgical time-out has been initiated not being stored, transmit a notification to a display included in the operating room including a prompt to perform the surgical time-out.