IP Library › Granted Patent US 12,207,887
Granted Patent B1
US 12,207,887 · App. 16/947,810 · Granted Jan 28, 2025

Systems and methods for detecting delays during a surgical procedure

Inventors: Joëlle Barral (Mountain View, CA); Martin Habbecke (Palo Alto, CA); Xing Jin (San Jose, CA); Caitlin Donhowe (Mountain View, CA)
Assignee: VERILY LIFE SCIENCES LLC
A61B34/20A61B34/25A61B34/30A61B90/37G06T7/0012G06V20/20G06V20/41G08B5/36G08B21/182G16H50/20G16H70/20H04L51/046H04W4/14A61B2034/2055A61B2034/301A61B2034/742G06T2207/10016G06T2207/30004G06V2201/03
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,207,887
App. No.
16/947,810
Granted
Jan 28, 2025
Kind
B1
Abstract

Examples of systems and methods for detecting delays during a surgical procedure are disclosed. For example, one disclosed method includes receiving, a computing device, video of a robotic surgical procedure; determining, by the computing device, an estimated time-to-complete (“TTC”) the robotic surgical procedure based on the video; and in response to determining a deviation between the estimated TTC and an expected duration of the robotic surgical procedure exceeds a threshold, outputting an indication that the robotic surgical procedure is deviating from the expected duration.

Claims (54)

1. A method comprising:

receiving, by a computing device, video of a robotic surgical procedure;

determining, by the computing device, an estimated time-to-complete (“TTC”) the robotic surgical procedure based on the video; and

in response to determining that a deviation exceeds a threshold, the deviation between the estimated TTC and an expected duration of the robotic surgical procedure:

determining one or more frames in the video, the one or more frames corresponding to an initial time of the deviation between the estimated TTC and the expected duration;

outputting an indication that the robotic surgical procedure is deviating from the expected duration; and

outputting an identification of the one or more frames corresponding to the initial time of the deviation.

2. The method of claim 1 , further comprising:

generating metadata corresponding to the one or more frames indicating the initial time of the deviation; and

associating the metadata with the video.

3. The method of claim 2 , wherein the metadata comprises a bookmark.

4. The method of claim 2 , further comprising determining a step of the robotic surgical procedure corresponding to the one or more frames.

5. The method of claim 1 , wherein receiving the video occurs in real-time during the robotic surgical procedure.

6. The method of claim 5 , wherein outputting the indication comprises displaying a graphical indication of the deviation.

7. The method of claim 5 , wherein outputting the indication comprises transmitting a message to a remote computing device associated with medical personal, the message indicating the deviation.

8. The method of claim 7 , wherein the transmitting the message comprises sending a text message to a mobile device.

9. The method of claim 1 , wherein receiving the video comprises receiving the video from a storage device.

10. A non-transitory computer-readable medium comprising processor-executable instructions configured to cause a processor to:

receive video of a robotic surgical procedure;

determine an estimated time-to-complete (“TTC”) the robotic surgical procedure based on the video; and

in response to a determination that a deviation exceeds a threshold, the deviation between the estimated TTC and an expected duration of the robotic surgical procedure:

determine one or more frames in the video, the one or more frames corresponding to an initial time of the deviation between the estimated TTC and the expected duration;

output an indication that the robotic surgical procedure is deviating from the expected duration; and

outputting an identification of the one or more frames corresponding to the initial time of the deviation.

11. The non-transitory computer-readable medium of claim 10 , further comprising processor-executable instructions configured to cause a processor to:

generate metadata corresponding to the one or more frames indicating the initial time of the deviation; and

associate the metadata with the video.

12. The non-transitory computer-readable medium of claim 11 , wherein the metadata comprises a bookmark.

13. The non-transitory computer-readable medium of claim 11 , further comprising processor-executable instructions configured to cause a processor to determine a step of the robotic surgical procedure corresponding to the one or more frames.

14. The non-transitory computer-readable medium of claim 10 , further comprising processor-executable instructions configured to cause a processor to receive the video in real-time during the robotic surgical procedure.

15. The non-transitory computer-readable medium of claim 14 , further comprising processor-executable instructions configured to cause a processor to display a graphical indication of the deviation.

16. The non-transitory computer-readable medium of claim 14 , further comprising processor-executable instructions configured to cause a processor to transmit a message to a remote computing device associated with medical personal, the message indicating the deviation.

17. The non-transitory computer-readable medium of claim 16 , further comprising processor-executable instructions configured to cause a processor to send a text message to a mobile device.

18. The non-transitory computer-readable medium of claim 10 , further comprising processor-executable instructions configured to cause a processor to receive the video from a storage device.

19. A computing device comprising:

a non-transitory computer-readable medium; and

a processor communicatively coupled to the non-transitory computer-readable medium, the processor configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:

receive video of a robotic surgical procedure;

determine an estimated time-to-complete (“TTC”) the robotic surgical procedure based on the video; and

in response to a determination that a deviation exceeds a threshold, the deviation between the estimated TTC and an expected duration of the robotic surgical procedure:

determine one or more frames in the video, the one or more frames corresponding to an initial time of the deviation between the estimated TTC and the expected duration, the initial time of the deviation based on a difference between the estimated TTC and an expected TTC being within a threshold difference;

output an indication that the robotic surgical procedure is deviating from the expected duration; and

outputting an identification of the one or more frames corresponding to the initial time of the deviation.

20. The computing device of claim 19 , wherein the processor is configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to cause a processor to:

determine one or more frames in the video, the one or more frames corresponding to an initial time of the deviation between the estimated TTC and the expected duration;

generate metadata corresponding to the one or more frames indicating the initial time of the deviation; and

associate the metadata with the video.

21. The computing device of claim 20 , wherein the metadata comprises a bookmark.

22. The computing device of claim 20 , wherein the processor is configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to cause a processor to determine a step of the robotic surgical procedure corresponding to the one or more frames.

23. The computing device of claim 19 , wherein the processor is configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to cause a processor to receive the video in real-time during the robotic surgical procedure.

24. The computing device of claim 23 , wherein the processor is configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to cause a processor to display a graphical indication of the deviation.

25. The computing device of claim 23 , wherein the processor is configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to cause a processor to transmit a message to a remote computing device associated with medical personal, the message indicating the deviation.

26. The computing device of claim 25 , wherein the processor is configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to cause a processor to send a text message to a mobile device.

27. The computing device of claim 19 , wherein the processor is configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to cause a processor to receive the video from a storage device.

Assignments (2)
CHANGE OF ADDRESS Recorded Nov 19, 2024
From: VERILY LIFE SCIENCES LLC
To: VERILY LIFE SCIENCES LLC
Reel/Frame 069390/0656 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 19, 2023
From: BARRAL, JOËLLE; HABBECKE, MARTIN; JIN, XING; DONHOWE, CAITLIN
To: VERILY LIFE SCIENCES LLC
Reel/Frame 063376/0460 →
Continuity (1)
Provisional Application 62888773 · Aug 19, 2019
References Cited (41)
US 9171477B2 · Luo et al. · 2015 [cited by applicant]
US 9788907B1 · Alvi et al. · 2017 [cited by applicant]
US 9836654B1 · Alvi et al. · 2017 [cited by applicant]
US 20100088726A1 · Curtis et al. · 2010 [cited by applicant]
US 20110046476A1 · Cinquin et al. · 2011 [cited by applicant]
US 20140220527A1 · Li et al. · 2014 [cited by applicant]
US 20140286533A1 · Luo et al. · 2014 [cited by applicant]
US 20150005622A1 · Zhao et al. · 2015 [cited by applicant]
US 20170132785A1 · Wshah et al. · 2017 [cited by applicant]
US 20190069957A1 · Barral · 2019 [cited by examiner]
US 20190090969A1 · Jarc et al. · 2019 [cited by applicant]
US 20190188567A1 · Yao et al. · 2019 [cited by applicant]
US 20190223961A1 · Barral et al. · 2019 [cited by applicant]
US 20200126661A1 · Flexman et al. · 2020 [cited by applicant]
US 20200175265A1 · Schön et al. · 2020 [cited by applicant]
US 20200226751A1 · Jin et al. · 2020 [cited by applicant]
US 20200237452A1 · Wolf · 2020 [cited by examiner]
CN 109147254 · 2019 [cited by applicant]
WO 2016200887 · 2016 [cited by applicant]
WO 2017076221 · 2017 [cited by applicant]
WO 2017083768 · 2017 [cited by applicant]
WO 2018219551 · 2018 [cited by applicant]
U.S. Appl. No. 16/736,467 , Notice of Allowance, Mailed on Oct. 20, 2022, 11 pages. [cited by applicant]
Jin et al., “EndoRCN: Recurrent Convolutional Networks for Recognition of Surgical Workflow in Cholecystectomy Procedure Video”, Institute of Electrical and Electronics Engineers, 2016, 4 pages. [cited by applicant]
Haro et al., “Surgical Gesture Classification from Video Data”, International Conference on Medical Image Computing and Computer-Assisted Intervention, Part I, Springer, LNCS, vol. 7510, Oct. 2012, pp. 34-41. [cited by applicant]
Lin et al., “Automatic Detection and Segmentation of Robot-Assisted Surgical Motions”, International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, LNCS, vol. 3749, No. 9, Feb. 2005,… [cited by applicant]
Padoy et al., “A Boosted Segmentation Method for Surgical Workflow Analysis”, International Conference on Medical Image Computing and Computer-Assisted Intervention, Part I, Springer, LNCS, vol. 4791, Nov. 2007, pp. 102… [cited by applicant]
International Application No. PCT/US2020/012740, International Search Report and Written Opinion, Mailed on May 20, 2020, 17 pages. [cited by applicant]
International PCT/US2020/012740, “Invitation to Pay Additional Fees and, Where Applicable, Protest Fee”, Mar. 5, 2020, 2 pages. [cited by applicant]
Subetha et al., “A Survey on Human Activity Recognition from Videos”, International Conference on Information Communication and Embedded Systems (ICICES), Feb. 2016, 7 pages. [cited by applicant]
Tao et al., “Surgical Gesture Segmentation and Recognition”, International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, LNCS, vol. 8151, Sep. 2013, pp. 1-8. [cited by applicant]
Twinanda et al., “EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos”, IEEE Transactions on Medical Imaging, vol. 36, No. 1, 2016, pp. 1-11. [cited by applicant]
Volkov et al., “Machine Learning and Coresets for Automated Real-Time Video Segmentation of Laparoscopic and Robot-Assisted Surgery”, IEEE International Conference on Robotics and Automation (ICRA), 2017, 6 pages. [cited by applicant]
Zappella et al., “Surgical Gesture Classification from Video and Kinematic Data”, Medical Image Analysis, vol. 17, Oct. 2013, pp. 732-745. [cited by applicant]
U.S. Appl. No. 16/736,467, Non-Final Office Action, mailed Nov. 17, 2021, 26 pages. [cited by applicant]
Jin et al., “SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network”, IEEE Transactions on Medical Imaging, vol. 37, Issue 5, May 2018, pp. 1114-1126. [cited by applicant]
U.S. Appl. No. 16/736,467 , Final Office Action, Mailed on May 20, 2022, 35 pages. [cited by applicant]
Dergachyova et al., “Automatic Data-Driven Real-Time Segmentation and Recognition of Surgical Workflow”, International journal of computer assisted radiology and surgery, vol. 11, No. 6, Apr. 2016, 16 pages. [cited by applicant]
U.S. Appl. No. 18/154,070 , “Non-Final Office Action”, filed Oct. 13, 2023, 10 pages. [cited by applicant]
U.S. Appl. No. 18/154,070 , “Final Office Action”, filed Feb. 5, 2024, 7 pages. [cited by applicant]
U.S. Appl. No. 18/154,070 , “Notice of Allowance”, filed Jul. 9, 2024, 8 pages. [cited by applicant]