IP Library › Granted Patent US 12,329,484
Granted Patent B2
US 12,329,484 · App. 18/208,233 · Granted Jun 17, 2025

User-interface control using master controller

Inventors: Ashwinram Suresh (San Jose, CA); Joey Chau (Cupertino, CA)
Assignee: Intuitive Surgical Operations, Inc.
A61B34/37A61B34/25A61B34/74A61B34/76G06F3/048
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,329,484
App. No.
18/208,233
Granted
Jun 17, 2025
Kind
B2
Abstract

A system for controlling a user interface of a teleoperated surgical system, the system comprises a first master controller communicatively coupled to the teleoperated surgical system; and a display device communicatively coupled to the teleoperated surgical system and configured to display a graphical user interface; and wherein the first master controller is configured to transmit a first input signal to an interface controller, the first input signal caused by manual manipulation of the first master controller, the interface controller to use the first input signal to update a graphical user interface presented by the display device.

Claims (44)

1. A teleoperated surgical system comprising:

a display;

an input device;

one or more instruments;

a control system configured to:

cause the display to present a surgical scene depicting a surgical site;

operate the input device in a first mode to control movement of the one or more instruments in three-dimensions of free space within boundaries of the surgical site based on user inputs received via the input device and to provide feedback from the one or more instruments to the input device;

cause the display to present a graphical user interface that is overlayed on the surgical scene, wherein the graphical user interface comprises one or more user interface elements and a pointer for interacting with the one or more user interface elements; and

while the graphical user interface is presented at the display, operate the input device in a second mode to control movement of the pointer of the graphical user interface within a two-dimensional haptic surface oriented in the free space within boundaries of the surgical site to a viewing plane of the graphical user interface that is overlayed on the surgical scene based on user inputs received via the input device and to provide feedback to the input device based on the graphical user interface.

2. The teleoperated surgical system of claim 1 , wherein the control system is further configured to:

while operating the input device in the first mode, enable movement of the input device in three dimensions and generate instrument control signals to effect movement of the one or more instruments in response to movement of the input device in three dimensions; and

while operating the input device in the second mode, constrain movement of the input device within the two-dimensional haptic surface oriented at an offset with respect to the viewing plane and generate pointer control signals to effect movement of the pointer within the graphical user interface based on movement of the input device within the two-dimensional haptic surface.

3. The teleoperated surgical system of claim 1 , wherein providing feedback to the input device based on the graphical user interface comprises providing feedback based on a position of the pointer within the graphical user interface relative to a position of one of the one or more user interface elements.

4. The teleoperated surgical system of claim 1 , wherein providing feedback to the input device based on the graphical user interface comprises applying a force to nudge the input device in a direction that corresponds with moving the pointer towards one of the one or more user interface elements.

5. The teleoperated surgical system of claim 1 , wherein providing feedback to the input device based on the graphical user interface comprises applying a force to resist a user input to move the input device in a direction that corresponds with moving the pointer away from one of the one or more user interface elements.

6. The teleoperated surgical system of claim 1 , wherein providing feedback to the input device based on the graphical user interface comprises applying a force to resist a user input to move the input device except in a direction towards one of the one or more user interface elements.

7. The teleoperated surgical system of claim 1 , wherein the haptic plane is oriented at a dynamic angle offset to the viewing plane.

8. The teleoperated surgical system of claim 1 , wherein providing feedback to the input device based on the graphical user interface comprises providing feedback to the input device to provide a sensation of touching one of the one or more user interface elements.

9. The teleoperated surgical system of claim 1 , wherein the one or more user interface elements comprises a default user interface element; and

wherein providing feedback to the input device based on the graphical user interface comprises one or more of: (i) applying a force to nudge the input device in a direction that corresponds with moving the pointer towards the default user interface element, (ii) applying a force to resist a user input to move the input device in a direction that corresponds with moving the pointer away from the default user interface element, (iii) applying a force to resist user input to move the input device except in a direction towards the default user interface element, or (iv) applying haptic feedback to the input device to provide an indication that the graphical user interface includes the default user interface element.

10. The teleoperated surgical system of claim 1 , wherein responsive to changes in the viewing plane of the graphical user interface, the two-dimensional haptic surface is adjusted to maintain a substantial parallel orientation with the viewing plane.

11. A method of interactive control of a teleoperated surgical system, the method comprising:

causing a display of the teleoperated surgical system to present a surgical scene depicting a surgical site;

operating an input device of the teleoperated surgical system in a first mode to control movement of one or more instruments in three-dimensions of free space within boundaries of the surgical site of the teleoperated surgical system based on inputs received via the input device and to provide feedback from the one or more instruments to the input device;

causing the display to present a graphical user interface that is overlayed on the surgical scene, wherein the graphical user interface comprises one or more user interface elements and a pointer for interacting with the one or more user interface elements; and

while the graphical user interface is presented by the display, operating the input device in a second mode to control movement of the pointer of the graphical user interface within a two-dimensional haptic surface oriented in the free space within boundaries of the surgical site to a viewing plane of the graphical user interface that is overlayed on the surgical scene based on inputs received via the input device and to provide feedback to the input device based on the graphical user interface.

12. The method of claim 11 , further comprising:

while operating the input device in the first mode, enabling movement of the input device in three dimensions and generate instrument control signals to effect movement of the one or more instruments in response to movement of the input device in three dimensions; and

while operating the input device in the second mode, constraining movement of the input device within the two-dimensional haptic surface oriented at an offset with respect to the viewing plane and generate pointer control signals to effect movement of the pointer within the graphical user interface based on movement of the input device within the two-dimensional haptic surface.

13. The method of claim 11 , wherein providing feedback to the input device based on the graphical user interface comprises providing feedback based on a position of the pointer within the graphical user interface relative to a position of one of the one or more user interface elements.

14. The method of claim 11 , wherein providing feedback to the input device based on the graphical user interface comprises applying a force to nudge the input device in a direction that corresponds with moving the pointer towards one of the one or more user interface elements.

15. The method of claim 11 , wherein providing feedback to the input device based on the graphical user interface comprises applying a force to resist a user input to move the input device (i) in a direction that corresponds with moving the pointer away from one of the one or more user interface elements or (ii) except in a direction towards one of the one or more user interface elements.

16. The method of claim 11 , wherein the haptic plane is oriented at a dynamic angle offset to the viewing plane.

17. The method of claim 11 , wherein providing feedback to the input device based on the graphical user interface comprises providing feedback to the input device to provide a sensation of touching one of the one or more user interface elements.

18. The method of claim 11 , wherein the one or more user interface elements comprises a default user interface element; and

wherein providing feedback to the input device based on the graphical user interface comprises one or more of: (i) applying a force to nudge the input device in a direction that corresponds with moving the pointer towards the default user interface element, (ii) applying a force to resist a user input to move the input device in a direction that corresponds with moving the pointer away from the default user interface element, (iii) applying a force to resist user input to move the input device except in a direction towards the default user interface element, or (iv) applying haptic feedback to the input device to provide an indication that the graphical user interface includes the default user interface element.

19. The method of claim 11 , wherein responsive to changes in the viewing plane of the graphical user interface, the two-dimensional haptic surface is adjusted to maintain a substantial parallel orientation with the viewing plane.

20. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause a control system of a teleoperated surgical system to perform steps comprising:

causing a display of the teleoperated surgical system to present a surgical scene depicting a surgical site;

operating an input device of the teleoperated surgical system in a first mode to control movement of one or more instruments in three-dimensions of free space within boundaries of the surgical site of the teleoperated surgical system based on inputs received via the input device and to provide feedback from the one or more instruments to the input device;

causing the display to present a graphical user interface that is overlayed on the surgical scene, wherein the graphical user interface comprises one or more user interface elements and a pointer for interacting with the one or more user interface elements; and

while the graphical user interface is presented by the display, operating the input device in a second mode to control movement of the pointer of the graphical user interface within a two-dimensional haptic surface oriented in the free space within boundaries of the surgical site to a viewing plane of the graphical user interface based on inputs received via the input device and to provide feedback to the input device based on the graphical user interface;

causing the display to present a graphical user interface that is overlayed on the surgical scene, wherein the graphical user interface comprises one or more user interface elements and a pointer for interacting with the one or more user interface elements; and

while the graphical user interface is presented by the display, operating the input device in a second mode to control movement of the pointer of the graphical user interface within a two-dimensional haptic surface oriented in space of the display of the surgical scene to a viewing plane of the graphical user interface that is overlayed on the surgical scene based on inputs received via the input device and to provide feedback to the input device based on the graphical user interface.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 13, 2023
From: SURESH, ASHWINRAM; CHAU, JOEY
To: INTUITIVE SURGICAL OPERATIONS, INC.
Reel/Frame 063927/0411 →
Continuity (5)
Continuation 17463178 · Aug 31, 2021
Continuation 16153405 · Oct 5, 2018
Continuation 15526696
Provisional Application 62079398 · Nov 13, 2014
Related Publication 20230320799A1 · Oct 12, 2023
References Cited (112)
US 4942538A · Yuan et al. · 1990 [cited by applicant]
US 5625576A · Massie et al. · 1997 [cited by applicant]
US 5749362A · Funda · 1998 [cited by examiner]
US 5786805A · Barry · 1998 [cited by applicant]
US 5788688A · Bauer · 1998 [cited by examiner]
US 5799055A · Peshkin et al. · 1998 [cited by applicant]
US 6392675B1 · Becker et al. · 2002 [cited by applicant]
US 6424885B1 · Niemeyer et al. · 2002 [cited by applicant]
US 6493608B1 · Niemeyer · 2002 [cited by applicant]
US 6522906B1 · Salisbury, Jr · 2003 [cited by examiner]
US 6659926B2 · Haag · 2003 [cited by applicant]
US 6659939B2 · Moll et al. · 2003 [cited by applicant]
US 6671581B2 · Niemeyer et al. · 2003 [cited by applicant]
US 6714201B1 · Grinstein et al. · 2004 [cited by applicant]
US 6714939B2 · Saldanha et al. · 2004 [cited by applicant]
US 6799065B1 · Niemeyer · 2004 [cited by applicant]
US 7131073B2 · Rosenberg · 2006 [cited by examiner]
US 7206626B2 · Quaid, III · 2007 [cited by applicant]
US 7206627B2 · Abovitz · 2007 [cited by examiner]
US 7234937B2 · Sachdeva et al. · 2007 [cited by applicant]
US 8095200B2 · Quaid, III · 2012 [cited by examiner]
US 8359114B2 · Steingart et al. · 2013 [cited by applicant]
US 8398541B2 · Dimaio et al. · 2013 [cited by applicant]
US 8551084B2 · Hauck et al. · 2013 [cited by applicant]
US 8571628B2 · Kang et al. · 2013 [cited by applicant]
US 8657736B2 · Diolaiti et al. · 2014 [cited by applicant]
US 8696548B2 · Gilboa · 2014 [cited by applicant]
US 9266239B2 · Miller · 2016 [cited by applicant]
US 10123846B2 · Suresh et al. · 2018 [cited by applicant]
US 10786315B2 · Suresh et al. · 2020 [cited by applicant]
US 11135029B2 · Suresh et al. · 2021 [cited by applicant]
US 11723734B2 · Suresh · 2023 [cited by applicant]
US 20020033799A1 · Mallett et al. · 2002 [cited by applicant]
US 20030030621A1 · Rosenberg et al. · 2003 [cited by applicant]
US 20030220541A1 · Salisbury et al. · 2003 [cited by applicant]
US 20040091845A1 · Azerad et al. · 2004 [cited by applicant]
US 20050027397A1 · Niemeyer · 2005 [cited by applicant]
US 20050093847A1 · Altkorn et al. · 2005 [cited by applicant]
US 20060058616A1 · Marquart et al. · 2006 [cited by applicant]
US 20080033240A1 · Hoffman et al. · 2008 [cited by applicant]
US 20080168364A1 · Miller et al. · 2008 [cited by applicant]
US 20090012533A1 · Barbagli et al. · 2009 [cited by applicant]
US 20100073150A1 · Olson et al. · 2010 [cited by applicant]
US 20100177042A1 · Chen · 2010 [cited by applicant]
US 20100191100A1 · Anderson et al. · 2010 [cited by applicant]
US 20100234857A1 · Itkowitz et al. · 2010 [cited by applicant]
US 20100309122A1 · Abe et al. · 2010 [cited by applicant]
US 20100311028A1 · Bell, III et al. · 2010 [cited by applicant]
US 20110015569A1 · Kirschenman et al. · 2011 [cited by applicant]
US 20110050566A1 · Sawai · 2011 [cited by applicant]
US 20110066406A1 · Tsai et al. · 2011 [cited by applicant]
US 20110082587A1 · Ziaei et al. · 2011 [cited by applicant]
US 20110238010A1 · Kirschenman et al. · 2011 [cited by applicant]
US 20120001644A1 · Baarman et al. · 2012 [cited by applicant]
US 20120004894A1 · Butler et al. · 2012 [cited by applicant]
US 20120056840A1 · Benko et al. · 2012 [cited by applicant]
US 20120109152A1 · Quaid, III · 2012 [cited by applicant]
US 20120278711A1 · Altkorn et al. · 2012 [cited by applicant]
US 20130023899A1 · Green · 2013 [cited by applicant]
US 20130111322A1 · Marum et al. · 2013 [cited by applicant]
US 20130245375A1 · DiMaio et al. · 2013 [cited by applicant]
US 20140081455A1 · Goldberg et al. · 2014 [cited by applicant]
US 20140088941A1 · Banerjee · 2014 [cited by examiner]
US 20140187857A1 · Wilson et al. · 2014 [cited by applicant]
US 20140276938A1 · Hsu et al. · 2014 [cited by applicant]
US 20150153833A1 · Pinault et al. · 2015 [cited by applicant]
US 20150286295A1 · Pepe · 2015 [cited by applicant]
US 20160229052A1 · Touma et al. · 2016 [cited by applicant]
US 20210030491A1 · Suresh et al. · 2021 [cited by applicant]
US 20210393351A1 · Suresh · 2021 [cited by applicant]
WO WO2007030173A1 · 2007 [cited by applicant]
WO WO2010104753A1 · 2010 [cited by applicant]
WO WO2010147766A1 · 2010 [cited by applicant]
WO WO2015167531A2 · 2015 [cited by applicant]
WO WO2016077543A1 · 2016 [cited by applicant]
WO WO2016077552A1 · 2016 [cited by applicant]
Claims filed dated Oct. 8, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 5 pages. [cited by applicant]
Extended European Search Report for Application No. EP15858228.8, mailed on Jul. 11, 2018, 9 pages. [cited by applicant]
Extended European Search Report for Application No. EP21160293.3 mailed on Jul. 21, 2021, 10 pages. [cited by applicant]
Final Office Action mailed Dec. 10, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 09 pages. [cited by applicant]
Final Office Action mailed Sep. 15, 2020 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 07 pages. [cited by applicant]
International Preliminary Report on Patentability for Application No. PCT/US2015/060317, mailed on May 26, 2017, 8 pages. [cited by applicant]
International Preliminary Report on Patentability for Application No. PCT/US2015/060330, mailed on May 26, 2017, 12 pages. [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2015/060317, mailed on Jan. 27, 2016, 12 pages. [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2015/060330, mailed on Feb. 1, 2016, 14 pages. [cited by applicant]
Non Final Office Action mailed Apr. 4, 2019 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 20 pages. [cited by applicant]
Non Final Office Action mailed Feb. 6, 2018 for U.S. Appl. No. 15/526,696, filed May 12, 2017, 8 pages. [cited by applicant]
Non Final Office Action mailed Jul. 8, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 9 pages. [cited by applicant]
Non Final Office Action mailed Mar. 10, 2020 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 7 pages. [cited by applicant]
Non Final Office Action mailed Dec. 18, 2020 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 7 pages. [cited by applicant]
Non Final Office Action mailed Feb. 18, 2020 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 7 pages. [cited by applicant]
Non Final Office Action mailed Feb. 21, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 5 pages. [cited by applicant]
Notice of Allowance mailed Jun. 3, 2021 for U.S. Appl. No. 16/153,405, filed May 10, 2018, 8 pages. [cited by applicant]
Notice of Allowance mailed Oct. 16, 2019 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 11 pages. [cited by applicant]
Notice of Allowance mailed Dec. 19, 2019 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 2 pages. [cited by applicant]
Notice of Allowance mailed Jul. 22, 2020 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 5 pages. [cited by applicant]
Notice of Allowance mailed Jun. 22, 2018 for U.S. Appl. No. 15/526,696, filed May 12, 2017, 7 pages. [cited by applicant]
Notice of Allowance mailed Jan. 27, 2020 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 5 pages. [cited by applicant]
Preliminary Amendment mailed Dec. 21, 2018 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 12 pages. [cited by applicant]
Response filed May 7, 2018 to Non Final Office Action mailed Feb. 6, 2018 for U.S. Appl. No. 15/526,696, filed May 12, 2017, 18 pages. [cited by applicant]
Response filed Oct. 8, 2019 to Non Final Office Action mailed Jul. 8, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 12 pages. [cited by applicant]
Response filed Jul. 11, 2019 to Non Final Office Action mailed Apr. 4, 2019 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 9 pages. [cited by applicant]
Response filed Feb. 17, 2020 to Final Office Action mailed Dec. 10, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 12 pages. [cited by applicant]
Response filed May 18, 2020 to Non Final Office Action mailed Feb. 18, 2020 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 9 pages. [cited by applicant]
Response filed Jul. 23, 2020 to Non Final Office Action mailed Mar. 10, 2020 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 10 pages. [cited by applicant]
Response filed Apr. 29, 2019 to Non Final Office Action mailed Feb. 21, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 9 pages. [cited by applicant]
Vertut, J, and Coiffet, P., “Robot Technology: Teleoperation and Robotics Evolution and Development,” English translation, Prentice-Hall, Inc., Inglewood Cliffs, Nj, USA 1986, vol. 3A, 332 pages. [cited by applicant]
Vogel, D. and Balakrishnan, R., “Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays,” Department of Computer Science—University of Toronto, Oct. 2005, pp. 33-42. [cited by applicant]
Chinese Application Serial No. 201580069917.X, Response filed Aug. 23, 2024 to Notice of Reexamination mailed Jul. 14, 2023, 21 Pages. [cited by applicant]
European Application Serial No. 21160293.3, Communication Pursuant to Article 94(3) EPC mailed Jun. 25, 2024, 8 Pages. [cited by applicant]
“European Application Serial No. 21160293.3, Response filed Oct. 31, 2024 to Communication Pursuant to Article 94(3) EPC mailed Jun. 25, 2024”, with English claims, 11 pages. [cited by applicant]
Korean Application Serial No. 10-2022-7044344, Response filed Aug. 30, 2024 to Final Office Action mailed May 31, 2024, 11 Pages. [cited by applicant]
Cited By (1)
US 1,139,401