Mobile virtual reality system for surgical robotic systems
Mobile virtual reality system for simulation, training or demonstration of a surgical robotic system can include a virtual reality processor. The processor can generate a virtual surgical robot and render the virtual surgical robot on a display. The virtual surgical robot can include a virtual surgical tool. A handheld user input device (UID) can sense a hand input from a hand. A foot input device can sense a foot input from a foot. The virtual reality processor can be configured to control a movement or action of the virtual surgical robot based on the hand input, and change which of the virtual surgical instruments is controlled by the one or more handheld UIDs based on the foot input. Other embodiments and aspects are disclosed and claimed.
1. A method performed by at least one programmed processor of a mobile device for providing an immersive mobile interface for simulation and teleoperation of a surgical robotic system, comprising:
receiving log-in information from a user;
syncing a user profile associated with the user with the log-in information to initiate a session, wherein the user profile has a first set of reconfigurable settings of a first surgical robotic procedure and a second set of reconfigurable settings of a second surgical robotic procedure;
displaying, on a display of the mobile device, a first selectable exercise for a simulation of the first surgical robotic procedure and a second selectable exercise for a live teleoperation of the second surgical robotic procedure; and
responsive to receiving a user-selection of the first selectable exercise, simulating, in the session, the first surgical robotic procedure according to the first set of reconfigurable settings by:
receiving user tracking data from a tracking system of the mobile device, and
manipulating a virtual robotic component associated with the first surgical robotic procedure based only on the user tracking data from the tracking system of the mobile device.
2. The method of claim 1 , wherein each of the first and second sets of reconfigurable settings includes one or more of the following:
a surgical robotic arm or tool type,
one or more metrics to measure during the exercise,
a tissue property,
a visual, auditory, haptic, or tactile feedback type, or
patient model settings.
3. The method of claim 1 further comprising, upon completion of the simulated first surgical robotic procedure:
determining a score of the simulated first surgical robotic procedure; and
displaying, on the display, feedback of the score to the user.
4. The method of claim 1 further comprising providing visual, auditory, haptic, or tactile feedback to the user during the simulated first surgical robotic procedure, the feedback being provided for one or more of the following: a workspace, faults, collisions, warnings, depth, tissue properties or interactions.
5. The method of claim 1 , wherein the tracking system comprises a camera and the user tracking data comprises image data of a hand or a handheld user interface device held in the user's hand.
6. The method of claim 1 , wherein the mobile device is a headworn device.
7. The method of claim 6 further comprising, responsive to receiving a user-selection of the second selectable exercise:
establishing, by the mobile device, the live teleoperation of the second surgical robotic procedure according to the second set of reconfigurable settings;
receiving, by the mobile device, user input via a handheld user input device (UID) to move a robotic arm of the surgical robotic system; and
transmitting, over a computer network and by the mobile device, a control signal to the robotic arm based on the user input.
8. The method of claim 1 further comprising, responsive to receiving a user-selection of the first selectable exercise,
displaying, on the display, one or more reconfigurable settings of the first set of reconfigurable settings of the first selectable exercise; and
receiving at least one user-selection of the one or more reconfigurable settings, wherein the first surgical robotic procedure is simulated based on the selected one or more reconfigurable settings.
9. A mobile device comprising:
a display;
a tracking system;
one or more processors; and
memory having stored therein instructions which when executed by the one or more processors causes the mobile device to:
receive log-in information from a user,
sync a user profile associated with the user with the log-in information to initiate a session, wherein the user profile has a first set of reconfigurable settings of a first surgical robotic procedure and a second set of reconfigurable settings of a second surgical robotic procedure,
display, on the display, a first selectable exercise for a simulation of the first surgical robotic procedure and a second selectable exercise for a live teleoperation of the second surgical robotic procedure, and
responsive to receiving a user-selection of the first selectable exercise, simulate, in the session, the first surgical robotic procedure according to the first set of reconfigurable settings by:
receiving user tracking data from the tracking system, and
manipulating a virtual robotic component associated with the first surgical robotic procedure based only on the user tracking data from the tracking system.
10. The mobile device of claim 9 , wherein each of the first and second sets of reconfigurable settings includes one or more of the following:
a surgical robotic arm or tool type,
one or more metrics to measure during the exercise,
a tissue property,
a visual, auditory, haptic, or tactile feedback type, or
patient model settings.
11. The mobile device of claim 9 , wherein the memory has further instructions to, upon completion of the simulated first surgical robotic procedure:
determine a score of the simulated first surgical robotic procedure; and
display, on the display, feedback of the score to the user.
12. The mobile device of claim 9 , wherein the memory has further instructions to provide visual, auditory, haptic, or tactile feedback to the user during the simulated first surgical robotic procedure, the feedback being provided for one or more of the following: a workspace, faults, collisions, warnings, depth, tissue properties or interactions.
13. The mobile device of claim 9 , wherein the memory has further instructions to update at least one reconfigurable setting of the first set of reconfigurable settings of the first surgical robotic procedure based on the session.
14. The mobile device of claim 9 , wherein the memory has further instructions to, responsive to receiving a user-selection of the second selectable exercise,
establish the live teleoperation of the second surgical robotic procedure according to the second set of reconfigurable settings;
receive user input via a handheld user input device (UID) to move a surgical robotic arm; and
transmit, over a computer network, a control signal to the robotic arm based on the user input.
15. The mobile device of claim 9 , wherein the memory has further instructions to, responsive to receiving a user-selection of the first selectable exercise,
display, on the display, one or more reconfigurable settings of the first set of reconfigurable settings of the first selectable exercise; and
receive at least one user-selection of the one or more reconfigurable settings, wherein the first surgical robotic procedure is simulated based on the selected one or more reconfigurable settings.
16. A mobile virtual reality system, comprising:
at least one processor;
a display; and
an inside-out tracking system for producing user tracking data,
wherein the at least one processor is configured to:
receive log-in information from a user,
sync a user profile associated with the user with the log-in information to initiate a session, wherein the user profile has a first set of reconfigurable settings of a first surgical robotic procedure and a second set of reconfigurable settings of a second surgical robotic procedure,
display, on the display, a first selectable exercise for a simulation of the first surgical robotic procedure and a second selectable exercise for a live teleoperation of the second surgical robotic procedure, and
responsive to receiving a user-selection of the first selectable exercise, simulate, in the session, the first surgical robotic procedure according to the first set of reconfigurable settings by:
receiving the user tracking data from the inside-out tracking system, and
manipulating a virtual robotic component associated with the first surgical robotic procedure based only on the user tracking data.
17. The mobile virtual reality system of claim 16 , wherein each of the first and second sets of reconfigurable settings includes one or more of the following:
a surgical robotic arm or tool type,
one or more metrics to measure during the exercise,
a tissue property,
a visual, auditory, haptic, or tactile feedback type, or
patient model settings.
18. The mobile virtual reality system of claim 16 has further instructions to, responsive to receiving a user-selection of the second selectable exercise,
establish the live teleoperation of the second surgical robotic procedure according to the second set of reconfigurable settings;
receive user input via a handheld user input device (UID) to move a surgical robotic arm; and
transmit, over a computer network, a control signal to the robotic arm based on the user input.
19. The mobile virtual reality system of claim 16 has further instructions to, responsive to receiving a user-selection of the first selectable exercise,
display, on the display, one or more reconfigurable settings of the first set of reconfigurable settings of the first selectable exercise; and
receive at least one user-selection of the one or more reconfigurable settings, wherein the first surgical robotic procedure is simulated based on the selected one or more reconfigurable settings.
20. The mobile virtual reality system of claim 16 further comprises:
a mobile device that includes the at least one processor and the display such that the at least one processor performs operations on the mobile device; and
a handheld user input device that includes the inside-out tracking system.