Handheld user interface device for a surgical robot
Disclosed herein is a mobile interface device to control a robotically-assisted surgical system. The mobile interface device provides a surgeon with a direct view of the surgical robotic system and allows a surgeon to easily and intuitively select, control, or manipulate various target components of the surgical robotic system. The mobile interface device may capture a live image of the surgical robotic system to automatically identify a robotic arm that appears in the center of the captured live image as the target component selected by the surgeon. Based on the current pose or position of the target component, the mobile interface device may generate a list of target poses and control options. The surgeon may select a control option to select a target pose and to manipulate the target component to command a robotically-assisted movement of the selected robotic arm from the current pose to the target pose.
1. A mobile device for a surgical robotic system, the mobile device comprising:
a touchscreen;
a camera configured to capture a live image of the surgical robotic system including a target component; and
a processor configured to:
display the live image on the touchscreen;
process the captured live image to automatically identify the target component and identify one or more target positions for the target component;
display a number of trajectories of the target component to the one or more target positions for the target component as an overlay to the target component;
receive a user selection from the touchscreen of one of the target positions as a desired target position for the target component and a user selection of one of the number of trajectories as a desired trajectory; and
generate a command based on the selected desired target position and selected desired trajectory to drive the target component to the desired target position, wherein the target component comprises a surgical table or a robotic arm, and the command drives a gear, a linkage, or a joint of the robotic arm, or a drive mechanism of the table, that moves the robotic arm or the table to the desired target position along the desired trajectory.
2. The mobile device of claim 1 , further configured to:
identify a plurality of control options to allow a user selection of a movement of the target component to the desired target position.
3. The mobile device of claim 2 , wherein to generate the command, the mobile device is configured to:
receive a continuous user activation of one of the plurality of control options; and
move the target component to the desired target position.
4. The mobile device of claim 1 , further configured to:
desired target trajectory selected by the user so that the desired target trajectory avoids colliding with or interfering with an operation of a second component of the surgical robotic system.
5. The mobile device of claim 4 , wherein if it is determined that the desired trajectory requires moving the second component to avoid colliding or interfering with operations of the target component, the processor is further configured to display one or more planned trajectories for moving the second component.
6. The mobile device of claim 1 ,
wherein the overlay comprises virtual information displayed on the captured live image to present an augmented reality view of the surgical robotic system.
7. The mobile device of claim 1 , wherein the captured live image includes a second component of the surgical robotic system, and wherein the mobile device is further configured to:
process the captured live image to identify a current position of the second component or a volume in space around the current position of the second component as one of the target positions of the target component.
8. The mobile device of claim 1 , wherein automatically identifying the target component comprises
processing the captured live image to determine three-dimensional positions relative to the mobile device of a plurality of surface points of the surgical robotic system; and
identify the target component from the three-dimensional positions of the plurality of surface points.
9. The mobile device of claim 1 , further configured to:
generate a second command to freeze a movement of the target component to the desired target position when the target component moves off the captured live image.
10. The mobile device of claim 1 , wherein automatically identifying the target component comprises
processing the captured live image to identify one or more possible target components; and
prevent one of the possible target components from being identified as the target component if the one possible target component moves off the captured live image.
11. The mobile device of claim 1 , further configured to:
process the captured live image to allow a user selection of a current position of a second component of the surgical robotic system as a desired target position of the target component.
12. A method for controlling a surgical robotic system, comprising:
capturing, using a mobile device, a live image of the surgical robotic system that includes a target component;
identifying, by the mobile device, the target component and one or more target positions for the target component from the captured live image;
displaying, on a touch screen, the captured live image including the target component and a number of trajectories of the target component to the one or more target positions for the target component as an overlay to the target component;
receiving, by the mobile device from a user selection on the touch screen, one of the target positions as a desired target position for the target component and a user selection of one of the number of trajectories as a desired trajectory; and
generating a command based on the selected desired target position and selected desired trajectory to drive the target component to the desired target position using the desired trajectory, wherein the target component comprises a surgical table or a robotic arm; and
driving, based on the command, a gear, a linkage, or a joint of the robotic arm, or a drive mechanism of the table, that moves the robotic arm or the table to the desired target position.
13. The method of claim 12 , further comprising:
identifying, on the mobile device, a plurality of control options to allow a user selection of a movement of the target component to the desired target position.
14. The method of claim 12 , further comprising:
calculating the desired target trajectory selected by the user so that the desired target trajectory avoids colliding with or interfering with an operation of a second component of the surgical robotic system; and
generating the command to drive the target component to the desired target position along the desired trajectory.
15. The method of claim 14 , wherein:
if it is determined that the desired trajectory requires moving the second component to avoid colliding or interfering with operations of the target component of the surgical robotic system, displaying one or more planned trajectories for moving the second component.
16. The method of claim 12 , wherein the overlay comprises virtual information of the target component; and
presenting, on the mobile device, an augmented reality view of the surgical robotic system comprising the overlay on the live image of the target component.
17. The method of claim 12 , further comprising:
identifying a second component of the surgical robotic system from the captured live image; and
identifying a current position of the second component or a volume in space around the current position of the second component as one of the target positions of the target component.
18. The method of claim 12 , further comprising:
generating a second command to freeze a movement of the target component to the desired target position when the target component moves off the captured live image.
19. The method of claim 12 , further comprising:
presenting, on the mobile device, the captured live image to allow a user selection of the target component or to allow a user selection of a current position of a second component of the surgical robotic system in the captured live image as a desired target position of the target component.
20. A non-transitory computer-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
capturing a live image of a surgical robotic system that includes a target component;
displaying, on a touchscreen, the captured live image;
identifying automatically the target component from the captured live image; identifying one or more target positions of the target component from the captured live image;
display a number of trajectories of the target component to the one or more target positions for the target component as an overlay to the target component;
receiving a user selection from the touchscreen of one of the target positions as a desired target position of the target component and a user selection of one of the number of trajectories as a desired trajectory; and
generating a command based on the selected desired target position and selected desired trajectory to drive the target component to the desired target position along the desired trajectory in response to the user selection, wherein the target component comprises a surgical table or a robotic arm, and the command drives a gear, a linkage, or a joint of the robotic arm, or a drive mechanism of the table, that moves the robotic arm or elevate the table to the desired target position.