IP Library › Granted Patent US 12,336,773
Granted Patent B2
US 12,336,773 · App. 17/016,340 · Granted Jun 24, 2025

Interaction between user-interface and master controller

Inventors: Ashwinram Suresh (San Jose, CA); Joey Chau (Cupertino, CA)
Assignee: Intuitive Surgical Operations, Inc.
A61B34/25A61B34/30A61B34/35A61B34/37A61B34/74A61B34/76A61B90/37G06F3/04812A61B2017/00203A61B2017/00207A61B2017/00216A61B2034/741A61B2034/742A61B2034/743A61B2034/744A61B90/03A61B2090/365
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,336,773
App. No.
17/016,340
Granted
Jun 24, 2025
Kind
B2
Abstract

A system to control the interaction between a user-interface of a teleoperated surgical system and an input device of the teleoperated surgical system, the system comprising a first master controller communicatively coupled to the teleoperated surgical system, a feedback control communicatively coupled to the first master controller, and a display device communicatively coupled to the teleoperated surgical system and configured to display the user interface. The feedback control is configured to restrict the movement of the first master controller based on a state of the user interface changing from a previous state.

Claims (57)

1. A method of interactive control of a teleoperated surgical system, the method comprising:

displaying at the teleoperated surgical system, a two-dimensional graphical user interface, wherein the two-dimensional graphical user interface includes (i) a graphical pointer that is moveable in two dimensions within the two-dimensional graphical user interface, (ii) a clickable element configured to be interacted with using the graphical pointer, and (iii) a user viewing plane;

constraining movement of a first master controller from a three-dimensional haptic space to a two-dimensional haptic plane oriented based on the user viewing plane of the two-dimensional graphical user interface;

receiving, at the teleoperated surgical system, sensor input indicating the two-dimensional movement of the first master controller constrained to the two-dimensional haptic plane, the sensor input to control movement of the graphical pointer, to move the graphical pointer within the two-dimensional graphical user interface correspondingly with the movement of the first master controller within the two-dimensional haptic plane;

in response to detecting a user intent to interact with the clickable element using the graphical pointer, further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer within the two-dimensional graphical user interface; and

detecting the user intent to interact with the clickable element using the graphical pointer based upon a change in speed of closing of a pincher formation of the first master controller.

2. The method of claim 1 , further including:

detecting the user intent to interact with the clickable element based upon a position of the graphical pointer in relation to the clickable element in the two-dimensional graphical user interface.

3. The method of claim 1 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer within the two-dimensional graphical user interface includes constraining the movement of the first master controller in the two-dimensional haptic plane to reduce the corresponding movement of the graphical pointer within the two-dimensional graphical user interface.

4. The method of claim 1 , further including:

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer within the two-dimensional graphical user interface includes at least partially locking the first master controller to stop the corresponding movement of the graphical pointer within the two-dimensional graphical user interface.

5. The method of claim 1 , further including:

detecting the user intent to interact with the clickable element in response to the graphical pointer hovering over the clickable element in the two-dimensional graphical user interface;

further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer within the two-dimensional graphical user interface while the graphical pointer hovers over the clickable element.

6. The method of claim 1 , further including:

detecting the user intent to interact with the clickable element based upon an amount that the pincher formation of the first master controller is closed.

7. The method of claim 6 , further including:

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes slowing the movement of the graphical pointer as a function of change in the amount that the pincher formation of the first master controller is closed.

8. The method of claim 6 , further including:

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes limiting the movement of the graphical pointer as a function of change in the amount that the pincher formation of the first master controller is closed.

9. The method of claim 8 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes decreasing the movement of the graphical pointer as the function of change in the amount that the pincher formation of the first master controller is closed in at least one of a linear, exponential, or logarithmic manner.

10. The method of claim 1 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes slowing the movement of the graphical pointer as a function of the change in speed of closing of the pincher formation of the first master controller.

11. The method of claim 1 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes limiting the movement of the graphical pointer as a function of the change in speed of closing of the pincher formation of the first master controller.

12. A control system to interactively control a teleoperated surgical system, the control system comprising:

a first master controller communicatively coupled to the teleoperated surgical system;

a display device communicatively coupled to the teleoperated surgical system;

wherein the control system is configured to:

display at the display device a two-dimensional graphical user interface, wherein the two-dimensional graphical user interface includes (i) a graphical pointer that is moveable in two dimensions within the two-dimensional graphical user interface, (ii) a clickable element configured to be interacted with using the graphical pointer, and (iii) a user viewing plane;

constrain movement of the first master controller from a three-dimensional haptic space to a two-dimensional haptic plane oriented based on the user viewing plane of the two-dimensional graphical user interface;

receive sensor input indicating the two-dimensional movement of the first master controller constrained to the two-dimensional haptic plane, the sensor input to control movement of the graphical pointer, to move the graphical pointer within the two-dimensional graphical user interface correspondingly with the movement of the first master controller within the two-dimensional haptic plane;

in response to detecting a user intent to interact with the clickable element using the graphical pointer, further constrain the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer within the two-dimensional graphical user interface; and

detect the user intent to interact with the clickable element using the graphical pointer based upon a change in speed of closing of a pincher formation of the first master controller.

13. The control system of claim 12 , wherein the control system is further configured to,

detect the user intent to interact with the clickable element based upon a position of the graphical pointer in relation to the clickable element in the two-dimensional graphical user interface.

14. The control system of claim 12 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer within the two-dimensional graphical user interface includes constraining the movement of the first master controller in the two-dimensional haptic plane to reduce the corresponding movement of the graphical pointer within the two-dimensional graphical user interface.

15. The control system of claim 12 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer within the two-dimensional graphical user interface includes at least partially locking the first master controller to stop the corresponding movement of the graphical pointer within the two-dimensional graphical user interface.

16. The control system of claim 12 , wherein the control system is further configured to,

detect the user intent to interact with the clickable element in response to the graphical pointer hovering over the clickable element in the two-dimensional graphical user interface;

further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer within the two-dimensional graphical user interface while the graphical pointer hovers over the clickable element.

17. The control system of claim 12 , wherein the control system is further configured to,

detect the user intent to interact with the clickable element based upon an amount that the pincher formation of the first master controller is closed.

18. The control system of claim 17 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes slowing the movement of the graphical pointer as a function of change in the amount that the pincher formation of the first master controller is closed.

19. The control system of claim 17 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes limiting the movement of the graphical pointer as a function of change in the amount that the pincher formation of the first master controller is closed.

20. The control system of claim 19 ,

Wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes decreasing the movement of the graphical pointer as the function of change in the amount that the pincher formation of the first master controller is closed in at least one of a linear, exponential, or logarithmic manner.

21. The control system of claim 12 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes slowing the movement of the graphical pointer as a function of the change in speed of closing of the pincher formation of the first master controller.

22. The control system of claim 12 ,

wherein further constraining the movement of the first master controller within the two-dimensional haptic plane to affect the movement of the graphical pointer includes limiting the movement of the graphical pointer as a function of the change in speed of closing of the pincher formation of the first master controller is.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 11, 2020
From: SURESH, ASHWINRAM; CHAU, JOEY
To: INTUITIVE SURGICAL OPERATIONS, INC.
Reel/Frame 053750/0160 →
Continuity (3)
Continuation 15526698
Provisional Application 62079402 · Nov 13, 2014
Related Publication 20210030491A1 · Feb 4, 2021
References Cited (114)
US 4942538A · Yuan et al. · 1990 [cited by applicant]
US 5625576A · Massie et al. · 1997 [cited by applicant]
US 5749362A · Funda et al. · 1998 [cited by applicant]
US 5786805A · Barry · 1998 [cited by examiner]
US 5788688A · Bauer et al. · 1998 [cited by applicant]
US 5799055A · Peshkin et al. · 1998 [cited by applicant]
US 6392675B1 · Becker · 2002 [cited by examiner]
US 6424885B1 · Niemeyer et al. · 2002 [cited by applicant]
US 6493608B1 · Niemeyer et al. · 2002 [cited by applicant]
US 6522906B1 · Salisbury, Jr. et al. · 2003 [cited by applicant]
US 6659926B2 · Haag · 2003 [cited by applicant]
US 6659939B2 · Moll et al. · 2003 [cited by applicant]
US 6671581B2 · Niemeyer et al. · 2003 [cited by applicant]
US 6714201B1 · Grinstein et al. · 2004 [cited by applicant]
US 6714939B2 · Saldanha et al. · 2004 [cited by applicant]
US 6799065B1 · Niemeyer · 2004 [cited by applicant]
US 7131073B2 · Rosenberg et al. · 2006 [cited by applicant]
US 7206626B2 · Quaid, III · 2007 [cited by applicant]
US 7206627B2 · Abovitz et al. · 2007 [cited by applicant]
US 7234937B2 · Sachdeva et al. · 2007 [cited by applicant]
US 8095200B2 · Quaid, III · 2012 [cited by applicant]
US 8359114B2 · Steingart et al. · 2013 [cited by applicant]
US 8398541B2 · Dimaio et al. · 2013 [cited by applicant]
US 8551084B2 · Hauck et al. · 2013 [cited by applicant]
US 8571628B2 · Kang et al. · 2013 [cited by applicant]
US 8657736B2 · Diolaiti et al. · 2014 [cited by applicant]
US 8696548B2 · Gilboa · 2014 [cited by applicant]
US 9266239B2 · Miller · 2016 [cited by applicant]
US 10123846B2 · Suresh et al. · 2018 [cited by applicant]
US 10786315B2 · Suresh et al. · 2020 [cited by applicant]
US 11135029B2 · Suresh et al. · 2021 [cited by applicant]
US 11723734B2 · Suresh · 2023 [cited by applicant]
US 20020033799A1 · Mallett · 2002 [cited by examiner]
US 20030030621A1 · Rosenberg · 2003 [cited by examiner]
US 20030220541A1 · Salisbury, Jr. · 2003 [cited by examiner]
US 20040091845A1 · Azerad et al. · 2004 [cited by applicant]
US 20050027397A1 · Niemeyer · 2005 [cited by examiner]
US 20050093847A1 · Altkorn et al. · 2005 [cited by applicant]
US 20060058616A1 · Marquart et al. · 2006 [cited by applicant]
US 20080033240A1 · Hoffman · 2008 [cited by examiner]
US 20080168364A1 · Miller · 2008 [cited by examiner]
US 20090012533A1 · Barbagli et al. · 2009 [cited by applicant]
US 20100073150A1 · Olson et al. · 2010 [cited by applicant]
US 20100177042A1 · Chen · 2010 [cited by examiner]
US 20100191100A1 · Anderson et al. · 2010 [cited by applicant]
US 20100234857A1 · Itkowitz et al. · 2010 [cited by applicant]
US 20100309122A1 · Abe · 2010 [cited by examiner]
US 20100311028A1 · Bell, III et al. · 2010 [cited by applicant]
US 20110015569A1 · Kirschenman et al. · 2011 [cited by applicant]
US 20110050566A1 · Sawai · 2011 [cited by examiner]
US 20110066406A1 · Tsai et al. · 2011 [cited by applicant]
US 20110082587A1 · Ziaei et al. · 2011 [cited by applicant]
US 20110238010A1 · Kirschenman et al. · 2011 [cited by applicant]
US 20120001644A1 · Baarman et al. · 2012 [cited by applicant]
US 20120004894A1 · Butler et al. · 2012 [cited by applicant]
US 20120056840A1 · Benko · 2012 [cited by examiner]
US 20120109152A1 · Quaid, III · 2012 [cited by examiner]
US 20120278711A1 · Altkorn et al. · 2012 [cited by applicant]
US 20130023899A1 · Green · 2013 [cited by applicant]
US 20130111322A1 · Marum · 2013 [cited by examiner]
US 20130245375A1 · Dimaio et al. · 2013 [cited by applicant]
US 20140081455A1 · Goldberg et al. · 2014 [cited by applicant]
US 20140088941A1 · Banerjee et al. · 2014 [cited by applicant]
US 20140187857A1 · Wilson et al. · 2014 [cited by applicant]
US 20140276938A1 · Hsu et al. · 2014 [cited by applicant]
US 20150153833A1 · Pinault · 2015 [cited by examiner]
US 20150286295A1 · Pepe · 2015 [cited by examiner]
US 20160229052A1 · Touma · 2016 [cited by examiner]
US 20170333139A1 · Suresh et al. · 2017 [cited by applicant]
US 20190099230A1 · Suresh et al. · 2019 [cited by applicant]
CN 118370610 · 2024 [cited by applicant]
WO WO2007030173A1 · 2007 [cited by applicant]
WO WO2010104753A1 · 2010 [cited by applicant]
WO WO2010147766A1 · 2010 [cited by applicant]
WO WO2015167531A2 · 2015 [cited by examiner]
WO WO2016077543A1 · 2016 [cited by applicant]
WO WO2016077552A1 · 2016 [cited by applicant]
Vogel eat al., “Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays”, UIST'05, Oct. 23-27, 2005, pp. 33-42. (Year: 2005). [cited by examiner]
Notice of Allowance mailed Jun. 3, 2021 for U.S. Appl. No. 16/153,405, filed May 10, 2018, 8 pages. [cited by applicant]
Non Final Office Action mailed Dec. 18, 2020 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 7 pages. [cited by applicant]
Claims filed dated Oct. 8, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 5 pages. [cited by applicant]
Extended European Search Report for Application No. EP15858228.8, mailed on Jul. 11, 2018, 9 pages. [cited by applicant]
Final Office Action mailed Dec. 10, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018. [cited by applicant]
International Preliminary Report on Patentability for Application No. PCT/US2015/060317, mailed on May 26, 2017, 8 pages. [cited by applicant]
International Preliminary Report on Patentability for Application No. PCT/US2015/060330, mailed on May 26, 2017, 12 pages. [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2015/060317, mailed on Jan. 27, 2016, 12 pages. [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2015/060330, mailed on Feb. 1, 2016. 14 pages. [cited by applicant]
Non Final Office Action mailed Apr. 4, 2019 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 20 pages. [cited by applicant]
Non Final Office Action mailed Feb. 6, 2018 for U.S. Appl. No. 15/526,696, filed May 12, 2017, 8 pages. [cited by applicant]
Non Final Office Action mailed Jul. 8, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 9 pages. [cited by applicant]
Non Final Office Action mailed Feb. 18, 2020 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 7 pages. [cited by applicant]
Non Final Office Action mailed Feb. 21, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 5 pages,. [cited by applicant]
Notice of Allowance mailed Oct. 16, 2019 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 11 pages. [cited by applicant]
Notice of Allowance mailed Dec. 19, 2019 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 2 pages. [cited by applicant]
Notice of Allowance mailed Jun. 22, 2018 for U.S. Appl. No. 15/526,696, filed May 12, 2017, 7 pages. [cited by applicant]
Notice of Allowance mailed Jan. 27, 2020 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 5 pages. [cited by applicant]
Preliminary Amendment mailed Dec. 21, 2018 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 12 pages. [cited by applicant]
Response filed May 7, 2018 to Non Final Office Action mailed Feb. 6, 2018 for U.S. Appl. No. 15/526,696, filed May 12, 2017, 18 pages. [cited by applicant]
Response filed Oct. 8, 2019 to Non Final Office Action mailed Jul. 8, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 12 pages. [cited by applicant]
Response filed Jul. 11, 2019 to Non Final Office Action mailed Apr. 4, 2019 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 9 pages. [cited by applicant]
Response filed Apr. 29, 2019 to Non Final Office Action mailed Feb. 21, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 9 pages. [cited by applicant]
Vertut, Jean and Phillipe Coiffet, Robot Technology: Teleoperation and Robotics Evolution and Development, English translation, Prentice-Hall, Inc., Inglewood Cliffs, NJ. USA 1986, vol. 3A, 332 pages. [cited by applicant]
Extended European Search Report for Application No. EP21160293.3 mailed on Jul. 21, 2021, 10 pages. [cited by applicant]
Final Office Action mailed Sep. 15, 2020 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018. [cited by applicant]
Non Final Office Action mailed Mar. 10, 2020 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 7 pages. [cited by applicant]
Notice of Allowance mailed Jul. 22, 2020 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 5 pages. [cited by applicant]
Response filed Feb. 17, 2020 to Final Office Action mailed Dec. 10, 2019 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 12 pages. [cited by applicant]
Response filed May 18, 2020 to Non Final Office Action mailed Feb. 18, 2020 for U.S. Appl. No. 15/526,698, filed May 12, 2017, 9 pages. [cited by applicant]
Response filed Jul. 23, 2020 to Non Final Office Action mailed Mar. 10, 2020 for U.S. Appl. No. 16/153,405, filed Oct. 5, 2018, 10 pages. [cited by applicant]
“European Application Serial No. 21160293.3, Communication Pursuant to Article 94(3) EPC mailed Jun. 25, 2024”, 8 pages. [cited by applicant]
“European Application Serial No. [Pending], Communication Pursuant to Article 94(3) EPC mailed Jun. 25, 2024”, 8 pages. [cited by applicant]
“Chinese Application Serial No. 201580069917.X, Response filed Aug. 23, 2024 to Notice of Reexamination mailed Jul. 14, 2023”, with English claims, 21 pages. [cited by applicant]
“Korean Application Serial No. 10-2022-7044344, Response filed Aug. 30, 2024 to Final Office Action mailed May 31, 2024”, with English claims, 11 pages. [cited by applicant]
“European Application Serial No. 21160293.3, Response filed Oct. 31, 2024 to Communication Pursuant to Article 94(3) EPC mailed Jun. 25, 2024”, with English claims, 11 pages. [cited by applicant]