Method and system for synchronizing playback of two recorded videos of the same surgical procedure
This disclosure provides techniques of synchronizing the playback of two recorded videos of the same surgical procedure. In one aspect, a process for generating a composite video from two recorded videos of a surgical procedure is disclosed. This process begins by receiving a first and second surgical videos of the same surgical procedure. The process then performs phase segmentation on each of the first and second surgical videos to segment the first and second surgical videos into a first set of video segments and a second set of video segments, respectively, corresponding to a sequence of predefined phases. Next, the process time-aligns each video segment of a given predefined phase in the first video with a corresponding video segment of the given predefined phase in the second video. The process next displays the time-aligned first and second surgical videos for comparative viewing.
1. A method performed by a programmed processor of an electronic device, comprising:
receiving a recorded video of a surgical procedure and a live video feed of a user performing the surgical procedure in a training session;
simultaneously playing back, through at least one display, the recorded video and the live video feed;
monitoring, during playback of the recorded video and the live video feed, a progression of the user performing the surgical procedure during the training session by determining whether the user is performing the surgical procedure faster or slower than the playback of the recorded video based on the live video feed; and
adjusting a playback speed of the recorded video or a frame rate of the recorded video by 1) increasing the playback speed of the recorded video or increasing the frame rate of the recorded video when the user is performing the surgical procedure faster than the playback of the recorded video or 2) reducing the playback speed of the recorded video or reducing the frame rate of the recorded video when the user is performing the surgical procedure slower than the playback of the recorded video.
2. The method of claim 1 ,
wherein the surgical procedure comprises a surgical task,
wherein monitoring the progression of the user comprises determining whether the user has completed the surgical task based on the live video feed, and
wherein adjusting playback comprises, responsive to determining that the recorded video has played back the surgical task while the user has not completed the surgical task, pausing the recorded video until the user has completed the surgical task.
3. The method of claim 2 ,
wherein during simultaneous playback the live video feed depicts the user performing the surgical task and the recorded video depicts a performance of the surgical task,
wherein determining whether the user has completed the surgical task comprises determining whether an event associated with the surgical task has occurred in the live video feed.
4. The method of claim 3 , wherein the surgical task is a first surgical task of the surgical procedure, wherein the method further comprises:
determining an occurrence of the event associated with the surgical task in the live video feed; and
continuing playback of the recorded video by playing back, through the at least one display, a second surgical task of the surgical procedure that is subsequent to the first surgical task.
5. The method of claim 1 , wherein the recorded video and the live video feed are displayed side-by-side on the at least one display or on two displays.
6. The method of claim 5 , wherein the at least one display is at least one of:
a single monitor;
a touchscreen; and
a virtual reality (VR) headset.
7. The method of claim 1 , wherein the recorded video is a training video performed by an expert of the surgical procedure, and the live video feed depicts the user replicating actions of the expert performing surgical tasks in the training video.
8. A system comprising:
at least one display;
at least one processor; and
memory having stored therein instructions which when executed by the at least one processor causes the system to:
receive a recorded video of a surgical procedure and a live video feed of a user performing the surgical procedure in a training session;
simultaneously playback, through the at least one display, the recorded video and the live video feed;
monitor, during playback of the recorded video and the live video feed, a progression of the user performing the surgical procedure during the training session by determining whether the user is performing the surgical procedure faster or slower than the playback of the recorded video based on the live video feed; and
adjust a playback speed of the recorded video or a frame rate of the recorded video by 1) increasing the playback speed of the recorded video or increasing the frame rate of the recorded video when the user is performing the surgical procedure faster than the playback of the recorded video or 2) reducing the playback speed of the recorded video or reducing the frame rate of the recorded video when the user is performing the surgical procedure slower than the playback of the recorded video.
9. The system of claim 8 ,
wherein the surgical procedure comprises a surgical task,
wherein the instructions to monitor the progression of the user comprises instructions to determine whether the user has completed the surgical task based on the live video feed, and
wherein the instructions to adjust playback comprises, responsive to determining that the recorded video has played back the surgical task while the user has not completed the surgical task, instructions to pause the recorded video until the user has completed the surgical task.
10. The system of claim 9 ,
wherein during simultaneous playback the live video feed depicts the user performing the surgical task and the recorded video depicts a performance of the surgical task,
wherein the instructions to determine whether the user has completed the surgical task comprises instructions to determine whether an event associated with the surgical task has occurred in the live video feed.
11. The system of claim 10 , wherein the surgical task is a first surgical task of the surgical procedure, wherein the memory has further instructions to:
determine an occurrence of the event associated with the surgical task in the live video feed; and
continue playback of the recorded video by playing back, through the at least one display, a second surgical task of the surgical procedure that is subsequent to the first surgical task.
12. The system of claim 8 , wherein the recorded video and the live video feed are displayed side-by-side on the at least one display or on two displays.
13. The system of claim 8 , wherein determining whether the user is performing the surgical procedure faster or slower comprises:
identifying a landmark event in the recorded video;
identifying a corresponding landmark event in the live video; and
determining whether the landmark event occurred after or before the corresponding landmark event occurred with respect to time.
14. The system of claim 8 , wherein the system receives the recorded video, simultaneously plays back the recorded video and the live video feed, monitors, and adjusts the playback speed of the recorded video or the frame rate of the recorded video in real-time.
15. A non-transitory machine-readable medium storing instructions executable by one or more processors of a system to cause the system to:
receive a recorded video of a surgical procedure and a live video feed of a user performing the surgical procedure in a training session;
simultaneously playback, through at least one display, the recorded video and the live video feed;
monitor, during playback of the recorded video and the live video feed, a progression of the user performing the surgical procedure during the training session by determining whether the user is performing the surgical procedure faster or slower than the playback of the recorded video based on the live video feed; and
adjust a playback speed of the recorded video or a frame rate of the recorded video by 1) increasing the playback speed of the recorded video or increasing the frame rate of the recorded video when the user is performing the surgical procedure faster than the playback of the recorded video or 2) reducing the playback speed of the recorded video or reducing the frame rate of the recorded video when the user is performing the surgical procedure slower than the playback of the recorded video.
16. The non-transitory machine-readable medium of claim 15 ,
wherein the surgical procedure comprises a surgical task,
wherein the instructions to monitor the progression of the user comprises instructions to determine whether the user has completed the surgical task based on the live video feed, and
wherein the instructions to adjust playback comprises, responsive to determining that the recorded video has played back the surgical task while the user has not completed the surgical task, instructions to pause the recorded video until the user has completed the surgical task.
17. The non-transitory machine-readable medium of claim 16 ,
wherein during simultaneous playback the live video feed depicts the user performing the surgical task and the recorded video depicts a performance of the surgical task,
wherein the instructions to determine whether the user has completed the surgical task comprises instructions to determine whether an event associated with the surgical task has occurred in the live video feed.
18. The non-transitory machine-readable medium of claim 17 , wherein the surgical task is a first surgical task of the surgical procedure, wherein the non-transitory machine-readable medium has further instructions to:
determine an occurrence of the event associated with the surgical task in the live video feed; and
continue playback of the recorded video by playing back, through the at least one display, a second surgical task of the surgical procedure that is subsequent to the first surgical task.
19. The non-transitory machine-readable medium of claim 15 , wherein determining whether the user is performing the surgical procedure faster or slower comprises:
identifying a landmark event in the recorded video;
identifying a corresponding landmark event in the live video; and
determining whether the landmark event occurred after or before the corresponding landmark event occurred with respect to time.
20. The non-transitory machine-readable medium of claim 15 , wherein the recorded video is a training video performed by an expert of the surgical procedure, and the live video feed depicts the user replicating actions of the expert performing surgical tasks in the training video.