Multi-panel graphical user interface for a robotic surgical system

1 . A robotic surgical system, comprising:

a robotic arm with an end effector;

an input device configured to control movement of the robotic arm with the end effector while a housing of the input device is in a hand of a user and while the robotic surgical system is in teleoperation;

an endoscopic camera configured to capture one or more endoscopic images of a surgical site during a surgical procedure that is performed while the robotic surgical system is in teleoperation;

a display configured to display a graphical user interface (GUI);

at least one processor; and

memory having instructions which when executed by the at least one processor causes the robotic surgical system to:

display the one or more endoscopic images in the GUI on the display;

determine whether a surgical task of the surgical procedure has been completed based on the one or more endoscopic images;

responsive to a determination that the surgical task is complete, switch the input device from controlling the movement of the robotic arm to controlling a user interface (UI) item in the GUI displayed on the display; and

receive, via the input device, an input to control the UI item, wherein the input includes a rotational movement of the housing of the input device that is received about a same axis as the end effector.

2 . The robotic surgical system of claim 1 , wherein the rotational movement is received about the same axis as the end effector such that the input device remains aligned with the end effector while the input device controls the UI item.

3 . The robotic surgical system of claim 1 , wherein the input is a first input, wherein the memory has further instructions to receive, via the input device, a second input that is different than the first input, wherein the switching of the input device from controlling the movement to controlling the UI item occurs and is maintained while the second input is received via the input device.

4 . The robotic surgical system of claim 3 , wherein the input device comprises a clutch mechanism, wherein the memory has further instructions to pause robotic arm movement while the second input is received via the clutch mechanism of the input device.

5 . The robotic surgical system of claim 1 , wherein the UI item comprises a menu with a plurality of selectable items that are arranged in a ring, wherein the rotational movement of the housing allows the user to navigate through the plurality of selectable items.

6 . The robotic surgical system of claim 1 , wherein the GUI includes a display panel that includes the one or more endoscopic images of the surgical site captured by the endoscopic camera during the surgical procedure, wherein the GUI further includes a floating UI item at an edge of the display panel that indicates that a surgical instrument is out of a field of view of the endoscopic camera and is at a location of the surgical site based on the edge at which the floating UI item is located.

7 . The robotic surgical system of claim 6 , wherein the floating UI item comprises a line that indicates an orientation of the endoscopic camera.

8 . The robotic surgical system of claim 1 , wherein the one or more endoscopic images of the surgical site includes a surgical instrument, and a floating UI item overlaid on top of the one or more endoscopic images, wherein the memory has further instructions to:

determine, based on the one or more endoscopic images, that the surgical instrument is moving within a field of view of the endoscopic camera; and

move the floating UI item to mirror the movement of the surgical instrument such that the floating UI item maintains a distance from the surgical instrument in the one or more endoscopic images.

9 . The robotic surgical system of claim 1 , wherein the one or more endoscopic images are displayed in a primary panel in the GUI, wherein the memory has further instructions to switch the one or more endoscopic images with visual content of a software application that is displayed in a secondary panel in the GUI such that the visual content is displayed in the primary panel and the one or more endoscopic images are displayed in the secondary panel, responsive to determining that the surgical task is complete.

10 . The robotic surgical system of claim 1 , wherein the instructions to determine that the surgical task is complete includes instructions to perform a machine vision algorithm upon the one or more endoscopic images that includes the surgical site to identify completion of the surgical task.

11 . A method performed by a robotic surgical system that includes a robotic arm with an end effector and a display, the method comprising:

displaying, in a graphical user interface (GUI) in the display, one or more endoscopic images captured by an endoscopic camera during a surgical procedure that is performed while the robotic surgical system is in teleoperation, wherein the one or more endoscopic images comprises a surgical site;

determining whether a surgical task of the surgical procedure has been completed based on the one or more endoscopic images;

responsive to determining that the surgical task is complete, switching an input device from controlling movement of the robotic arm to controlling a user interface (UI) item in the GUI displayed on the display; and

receiving, via the input device, an input to control the UI item, wherein the input includes a rotational movement of the input device that is received about a same axis as the end effector.

12 . The method of claim 11 , wherein the rotational movement is received about the same axis as the end effector such that the input device remains aligned with the end effector while the input device controls the UI item.

13 . The method of claim 11 , wherein the input is a first input, wherein the method further comprises receiving, via the input device, a second input that is different than the first input, wherein the switching of the input device from controlling the movement to controlling the UI item occurs and is maintained while the second input is received via the input device.

14 . The method of claim 13 , wherein the input device comprises a clutch mechanism, wherein the method further comprises pausing robotic arm movement while the second input is received via the clutch mechanism of the input device.

15 . The method of claim 11 , wherein the UI item comprises a menu with a plurality of selectable items that are arranged in a ring, wherein the rotational movement of the input device allows a user of the input device to navigate through the plurality of selectable items.

16 . The method of claim 11 , wherein the GUI includes a display panel that includes the one or more endoscopic images of the surgical site captured by the endoscopic camera during the surgical procedure, wherein the GUI further includes a floating UI item at an edge of the display panel that indicates that a surgical instrument is out of a field of view of the endoscopic camera and is at a location of the surgical site based on the edge at which the floating UI item is located.

17 . The method of claim 16 , wherein the floating UI item comprises a line that indicates an orientation of the endoscopic camera.

18 . The method of claim 11 , wherein the one or more endoscopic images of the surgical site includes a surgical instrument, and a floating UI item overlaid on top of the one or more endoscopic images, wherein the method further comprises:

determining, based on the one or more endoscopic images, that the surgical instrument is moving within a field of view of the endoscopic camera; and

moving the floating UI item to mirror the movement of the surgical instrument such that the floating UI item maintains a distance from the surgical instrument in the one or more endoscopic images.

19 . The method of claim 11 , wherein the one or more endoscopic images are displayed in a primary panel in the GUI, wherein the method further comprises switching the one or more endoscopic images with visual content of a software application that is displayed in a secondary panel in the GUI such that the visual content is displayed in the primary panel and the one or more endoscopic images are displayed in the secondary panel, responsive to determining that the surgical task is complete.

20 . The method of claim 11 , wherein determining that the surgical task is complete comprises performing a machine vision algorithm upon the one or more endoscopic images that includes the surgical site to identify completion of the surgical task.