IP Library › Granted Patent US 12,708,458
Granted Patent B2
US 12,708,458 · App. 18/479,813 · Granted Aug 18, 2026

Multi-panel graphical user interface for a robotic surgical system

Inventors: Eric Mark Johnson (Pacific Grove, CA); Emma Essock-Burns (Mountain View, CA); Lawrence Edward Miller (Scotts Valley, CA); Francois W. Brahic (San Francisco, CA)
Assignee: Auris Health, Inc.
A61B34/25A61B1/0005A61B34/30A61B34/74A61B90/361B25J9/161B25J9/1666B25J9/1689G06F3/0482G06F3/04845G09G5/14A61B1/00045A61B2017/00119A61B2017/00128A61B2017/00199A61B2017/00203A61B2017/00207A61B2017/00212A61B2017/00216A61B2017/00221A61B2017/00225A61B18/1482A61B2034/101A61B2034/252A61B2034/254A61B2034/256A61B2034/258A61B2034/302A61B34/35A61B2034/741A61B2034/743A61B2034/744A61B2090/065A61B2090/365A61B2090/372G05B2219/45118G05B2219/45119G09G2340/045G09G2340/0464G09G2354/00G09G2380/08Y10S901/06Y10S901/41
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,708,458
App. No.
18/479,813
Filed
Oct 2, 2023
Granted
Aug 18, 2026
Kind
B2
Art Unit
2144
USPC
715/761
Abstract

A method for a robotic surgical system includes displaying a graphical user interface on a display to a user, wherein the graphical user interface includes a plurality of reconfigurable display panels, receiving a user input at one or more user input devices, wherein the user input indicates a selection of at least one software application relating to the robotic surgical system, and rendering content from the at least one selected software application among the plurality of reconfigurable display panels.

Claims (38)

1 . A robotic surgical system, comprising:

a robotic arm with an end effector;

an input device configured to control movement of the robotic arm with the end effector while a housing of the input device is in a hand of a user and while the robotic surgical system is in teleoperation;

an endoscopic camera configured to capture one or more endoscopic images of a surgical site during a surgical procedure that is performed while the robotic surgical system is in teleoperation;

a display configured to display a graphical user interface (GUI);

at least one processor; and

memory having instructions which when executed by the at least one processor causes the robotic surgical system to:

display the one or more endoscopic images in the GUI on the display;

determine whether a surgical task of the surgical procedure has been completed based on the one or more endoscopic images;

responsive to a determination that the surgical task is complete, switch the input device from controlling the movement of the robotic arm to controlling a user interface (UI) item in the GUI displayed on the display; and

receive, via the input device, an input to control the UI item, wherein the input includes a rotational movement of the housing of the input device that is received about a same axis as the end effector.

2 . The robotic surgical system of claim 1 , wherein the rotational movement is received about the same axis as the end effector such that the input device remains aligned with the end effector while the input device controls the UI item.

3 . The robotic surgical system of claim 1 , wherein the input is a first input, wherein the memory has further instructions to receive, via the input device, a second input that is different than the first input, wherein the switching of the input device from controlling the movement to controlling the UI item occurs and is maintained while the second input is received via the input device.

4 . The robotic surgical system of claim 3 , wherein the input device comprises a clutch mechanism, wherein the memory has further instructions to pause robotic arm movement while the second input is received via the clutch mechanism of the input device.

5 . The robotic surgical system of claim 1 , wherein the UI item comprises a menu with a plurality of selectable items that are arranged in a ring, wherein the rotational movement of the housing allows the user to navigate through the plurality of selectable items.

6 . The robotic surgical system of claim 1 , wherein the GUI includes a display panel that includes the one or more endoscopic images of the surgical site captured by the endoscopic camera during the surgical procedure, wherein the GUI further includes a floating UI item at an edge of the display panel that indicates that a surgical instrument is out of a field of view of the endoscopic camera and is at a location of the surgical site based on the edge at which the floating UI item is located.

7 . The robotic surgical system of claim 6 , wherein the floating UI item comprises a line that indicates an orientation of the endoscopic camera.

8 . The robotic surgical system of claim 1 , wherein the one or more endoscopic images of the surgical site includes a surgical instrument, and a floating UI item overlaid on top of the one or more endoscopic images, wherein the memory has further instructions to:

determine, based on the one or more endoscopic images, that the surgical instrument is moving within a field of view of the endoscopic camera; and

move the floating UI item to mirror the movement of the surgical instrument such that the floating UI item maintains a distance from the surgical instrument in the one or more endoscopic images.

9 . The robotic surgical system of claim 1 , wherein the one or more endoscopic images are displayed in a primary panel in the GUI, wherein the memory has further instructions to switch the one or more endoscopic images with visual content of a software application that is displayed in a secondary panel in the GUI such that the visual content is displayed in the primary panel and the one or more endoscopic images are displayed in the secondary panel, responsive to determining that the surgical task is complete.

10 . The robotic surgical system of claim 1 , wherein the instructions to determine that the surgical task is complete includes instructions to perform a machine vision algorithm upon the one or more endoscopic images that includes the surgical site to identify completion of the surgical task.

11 . A method performed by a robotic surgical system that includes a robotic arm with an end effector and a display, the method comprising:

displaying, in a graphical user interface (GUI) in the display, one or more endoscopic images captured by an endoscopic camera during a surgical procedure that is performed while the robotic surgical system is in teleoperation, wherein the one or more endoscopic images comprises a surgical site;

determining whether a surgical task of the surgical procedure has been completed based on the one or more endoscopic images;

responsive to determining that the surgical task is complete, switching an input device from controlling movement of the robotic arm to controlling a user interface (UI) item in the GUI displayed on the display; and

receiving, via the input device, an input to control the UI item, wherein the input includes a rotational movement of the input device that is received about a same axis as the end effector.

12 . The method of claim 11 , wherein the rotational movement is received about the same axis as the end effector such that the input device remains aligned with the end effector while the input device controls the UI item.

13 . The method of claim 11 , wherein the input is a first input, wherein the method further comprises receiving, via the input device, a second input that is different than the first input, wherein the switching of the input device from controlling the movement to controlling the UI item occurs and is maintained while the second input is received via the input device.

14 . The method of claim 13 , wherein the input device comprises a clutch mechanism, wherein the method further comprises pausing robotic arm movement while the second input is received via the clutch mechanism of the input device.

15 . The method of claim 11 , wherein the UI item comprises a menu with a plurality of selectable items that are arranged in a ring, wherein the rotational movement of the input device allows a user of the input device to navigate through the plurality of selectable items.

16 . The method of claim 11 , wherein the GUI includes a display panel that includes the one or more endoscopic images of the surgical site captured by the endoscopic camera during the surgical procedure, wherein the GUI further includes a floating UI item at an edge of the display panel that indicates that a surgical instrument is out of a field of view of the endoscopic camera and is at a location of the surgical site based on the edge at which the floating UI item is located.

17 . The method of claim 16 , wherein the floating UI item comprises a line that indicates an orientation of the endoscopic camera.

18 . The method of claim 11 , wherein the one or more endoscopic images of the surgical site includes a surgical instrument, and a floating UI item overlaid on top of the one or more endoscopic images, wherein the method further comprises:

determining, based on the one or more endoscopic images, that the surgical instrument is moving within a field of view of the endoscopic camera; and

moving the floating UI item to mirror the movement of the surgical instrument such that the floating UI item maintains a distance from the surgical instrument in the one or more endoscopic images.

19 . The method of claim 11 , wherein the one or more endoscopic images are displayed in a primary panel in the GUI, wherein the method further comprises switching the one or more endoscopic images with visual content of a software application that is displayed in a secondary panel in the GUI such that the visual content is displayed in the primary panel and the one or more endoscopic images are displayed in the secondary panel, responsive to determining that the surgical task is complete.

20 . The method of claim 11 , wherein determining that the surgical task is complete comprises performing a machine vision algorithm upon the one or more endoscopic images that includes the surgical site to identify completion of the surgical task.

Assignments (1)
MERGER Recorded Jan 27, 2026
From: VERB SURGICAL INC.
To: AURIS HEALTH, INC.
Reel/Frame 073601/0736 →
Continuity (3)
Continuation 17359044 · Jun 25, 2021
Continuation 15842485 · Dec 14, 2017
Related Publication 20240099790A1 · Mar 28, 2024
References Cited (67)
US 6847336B1 · Lemelson et al. · 2005 [cited by applicant]
US 9788907B1 · Alvi · 2017 [cited by applicant]
US 10368955B2 · Cone et al. · 2019 [cited by applicant]
US 10661453B2 · Koenig et al. · 2020 [cited by applicant]
US 10786327B2 · Anderson et al. · 2020 [cited by applicant]
US 20040243147A1 · Lipow · 2004 [cited by applicant]
US 20050043719A1 · Sanchez et al. · 2005 [cited by applicant]
US 20090036902A1 · DiMaio et al. · 2009 [cited by applicant]
US 20090192524A1 · Itkowitz · 2009 [cited by applicant]
US 20090221907A1 · Bar-Tal · 2009 [cited by applicant]
US 20090248036A1 · Hoffman et al. · 2009 [cited by applicant]
US 20090326556A1 · Diolaiti · 2009 [cited by applicant]
US 20100228249A1 · Mohr et al. · 2010 [cited by applicant]
US 20110187707A1 · Kaufman · 2011 [cited by applicant]
US 20120035764A1 · Lipow et al. · 2012 [cited by applicant]
US 20150317068A1 · Marka · 2015 [cited by applicant]
US 20160015471A1 · Piron et al. · 2016 [cited by applicant]
US 20160242858A1 · Moctezuma de la Barrera · 2016 [cited by applicant]
US 20160249992A1 · Ruiz Morales et al. · 2016 [cited by applicant]
US 20170337027A1 · Chan · 2017 [cited by applicant]
US 20180078034A1 · Savall et al. · 2018 [cited by applicant]
US 20180185113A1 · Gregerson et al. · 2018 [cited by applicant]
US 20190151032A1 · Mustufa · 2019 [cited by applicant]
US 20190231459A1 · Mustufa · 2019 [cited by examiner]
US 20190254759A1 · Azizian · 2019 [cited by examiner]
US 20190333626A1 · Mansi · 2019 [cited by applicant]
US 20200038125A1 · Farlow · 2020 [cited by examiner]
US 20200163731A1 · Itkowitz · 2020 [cited by examiner]
US 20200170731A1 · Itkowitz · 2020 [cited by applicant]
US 20200222138A1 · Diolaiti · 2020 [cited by examiner]
US 20200405420A1 · Purohit · 2020 [cited by applicant]
US 20220160445A1 · Meglan · 2022 [cited by applicant]
CN 101095635A · 2008 [cited by applicant]
CN 101467894A · 2009 [cited by applicant]
CN 101530325A · 2009 [cited by applicant]
KR 1020120045734A · 2012 [cited by applicant]
WO 2005089065A2 · 2005 [cited by applicant]
WO 2010120407A1 · 2010 [cited by applicant]
WO 2014153396A1 · 2014 [cited by applicant]
Examiner's Report dated May 18, 2021 issued by the Canadian Patent Office for Patent Application No. 3,079,816. [cited by applicant]
U.S. Appl. No. 62/432,528, filed Dec. 9, 2016. [cited by applicant]
Examination Report No. 1 of the Australian Patent Office dated Sep. 11, 2020 for related Australian Patent Application No. 2017442686. [cited by applicant]
Extended European Search Report and Search Opinion of the European Patent Office dated Sep. 15, 2020 for related European Patent Application No. 17934821.4. [cited by applicant]
MiRai, “ISBoxer—Window Layouts—The Wizard” Available Online at <https://www.youtube.com/watch?v=pA3MqP6LAAg>, Nov. 23, 2011, 1 page. [cited by applicant]
Notice of Acceptance of the Australian Patent Office dated Oct. 13, 2020 for related Australian Patent Application No. 2017442686. [cited by applicant]
International Preliminary Report on Patentability for International Application No. PCT/US2017/066489 mailed Jun. 25, 2020, 6 pages. [cited by applicant]
PCT Search Report and Written Opinion dated Feb. 21, 2018 for related PCT Application No. PCT/US2017/066489. [cited by applicant]
Advisory Action of the U.S. Patent Office dated May 8, 2020 for related U.S. Appl. No. 15/842,485. [cited by applicant]
Final Office Action of the U.S. Patent Office dated Feb. 19, 2020 for related U.S. Appl. No. 15/842,485. [cited by applicant]
Non-Final Office Action of the U.S. Patent Office dated Dec. 9, 2019 for related U.S. Appl. No. 15/842,485. [cited by applicant]
Non-Final Office Action of the U.S. Patent Office dated Nov. 4, 2020 for related U.S. Appl. No. 15/842,485. [cited by applicant]
Notice of Allowance of the U.S. Patent Office dated Feb. 24, 2021 for related U.S. Appl. No. 15/842,485. [cited by applicant]
Notice of Allowance of the U.S. Patent Office dated Jun. 28, 2021 for related U.S. Appl. No. 15/842,485. [cited by applicant]
Notice of Reasons for Rejection of the Korean Patent Office dated Jul. 26, 2021 for related Korean Patent Application No. 10-2020-7015478. [cited by applicant]
Non-Final Office Action of the U.S. Patent Office dated Dec. 21, 2021 for related U.S. Appl. No. 17/359,044. [cited by applicant]
First Office Action of the Chinese Patent Office dated Dec. 30, 2021 for related Chinese Patent Application No. 201780004289.6. [cited by applicant]
Office Action of the Canadian Patent Office dated Jan. 25, 2022 for related Canadian Patent Application No. 3079816. [cited by applicant]
Final Notice of Preliminary Rejection of the Korean Intellectual Property Office dated Jan. 27, 2022 for related Korean Patent Application No. 10-2020-7015478. [cited by applicant]
Final Office Action of the U.S. Patent Office dated Jul. 11, 2022 for related U.S. Appl. No. 17/359,044. [cited by applicant]
Notice of Final Rejection of the Korean Patent Office dated Jul. 19, 2022 for related Korean Patent Application No. 10-2020-7015478. [cited by applicant]
Office Action of the Canadian Patent Office dated May 18, 2021 for related Canadian Patent Application No. 3079816. [cited by applicant]
Office Action of the Korean Patent Office dated Sep. 20, 2022 for related Korean Patent Application No. 10-2020-7015478. [cited by applicant]
Second Office Action of the Chinese Patent Office dated Jun. 13, 2022 for related Chinese Patent Application No. 201780004289.6. [cited by applicant]
Notice of Allowance dated Oct. 14, 22 for Canadian Patent Application No. 3079816. [cited by applicant]
Non-Final Office Action of the U.S. Patent Office dated Nov. 25, 2022 for related U.S. Appl. No. 17/359,044. [cited by applicant]
Notification to Grant Patent Right for Invention of the Chinese Patent Office dated Sep. 15, 2022 for related Chinese Patent Application No. 201780004289.6. [cited by applicant]
Notice of Allowance of the U.S. Patent Office dated Jun. 7, 2023 for related U.S. Appl. No. 17/359,044. [cited by applicant]