Virtual reality system for simulating surgical workflows with patient models

1. A method for planning surgical robotic workflow with a surgical robotic system, comprising:

generating a virtual surgical environment including a virtual robotic arm and a virtual patient that is generated from a three-dimensional scanning of a physical patient;

determining a size or location of a workspace inside the virtual patient based on a geometry of the virtual patient;

determining a position of a virtual tool, including a trocar and virtual end effector, in the virtual patient to maximize a reach of the virtual end effector in the workspace;

determining a position of the virtual robotic arm based on the position of the virtual tool to maintain the reach of the virtual end effector in the workspace, wherein the reach of the virtual end effector is limited by movement constraints of the virtual robotic arm on which the virtual tool is attached to; and

presenting to a display, the position of the virtual tool and the position of the virtual robotic arm for the planning of the surgical robotic workflow.

2. The method according to claim 1 , wherein the size or location of the workspace is further determined based on a procedure type.

3. The method according to claim 1 , wherein the position of the virtual tool is further based on kinematics parameters that define the movement constraints of the virtual robotic arm.

4. The method according to claim 3 , wherein the kinematics parameters are extracted from a computer model that defines geometry and movement constraints of a physical robotic arm.

5. The method according to claim 3 , wherein the kinematics parameters are determined based on motor control data of a physical robotic arm captured during a physical surgical procedure or physical simulation.

6. The method according to claim 3 , further comprising

determining a point of entry on the virtual patient based on the position of the virtual tool.

7. The method according to claim 6 , further comprising

determining a location or orientation of a virtual control tower or a virtual user console in the virtual surgical environment based on providing sufficient clearance for medical personnel and other medical equipment, and minimizing risk of collisions.

8. The method according to claim 7 , further comprising

repeating the method to update the virtual surgical environment based on received inputs that define changes to the virtual patient or a procedure.

9. The method of claim 1 , wherein operations are performed in a sequential order, including a) first, determining the workspace based on the virtual patient and surgical procedure type; b) second, determining the position of the virtual tool based on the workspace; c) third, determining the position of the virtual robotic arm based on the position of the virtual tool; and d) fourth, determining a location and orientation of remaining virtual robotic equipment.

10. The method according to claim 1 , further comprising displaying the virtual surgical environment as a stadium view or as a first-person view.

11. The method according to claim 1 , further comprising displaying the virtual surgical environment through a head-mounted display.

12. The method of claim 1 , wherein the virtual patient is generated or modified based on sensor information or surgical robotic movement data, captured from one or more physical surgical procedures.

13. The method according to claim 12 , wherein generating or modifying the virtual patient includes estimating internal volume of the virtual patient based on a) camera images internal of a physical patient, or b) surgical robotic tool motions, captured during the one or more physical surgical procedures.

14. The method according to claim 12 , wherein generating or modifying the virtual patient includes reconstructing internal volume of the virtual patient based on endoscope data captured during the one or more physical surgical procedures.

15. A system for planning surgical workflow in a surgical robotic system, comprising a virtual reality processor, configured to perform operations including

generating a virtual surgical environment, the virtual surgical environment including a virtual robotic arm and a virtual patient that is generated from a three-dimensional scanning of a physical patient;

determining a size or location of a workspace inside the virtual patient based on a geometry of the virtual patient;

determining a position of a virtual tool, including a trocar and a virtual end effector, in the virtual patient to maximize a reach of the virtual end effector in the workspace;

determining a position of the virtual robotic arm based on the position of the virtual tool to maintain the reach of the virtual end effector in the workspace, wherein the reach of the virtual end effector is limited by movement constraints of the virtual robotic arm on which the virtual tool is attached to; and

presenting to a display, the position of the virtual tool and the position of the virtual robotic arm for the planning of the surgical workflow.

16. The system according to claim 15 , wherein the operations are performed in an order based on importance, the order including a) determining the workspace based on the virtual patient and surgical procedure type; b) next, determining the position of the virtual tool based on the workspace; c) next, determining the position of the virtual robotic arm based on the position of the virtual tool; and d) next, determining a location and orientation of remaining virtual robotic equipment.

17. The system according to claim 15 , wherein the workspace is further determined based on an internal volume of the virtual patient, the internal volume being determined based on a) camera images internal of a physical patient, or b) motions of a physical surgical robotic arm or tool captured during the one or more physical surgical procedures.

18. The system, according to claim 15 , further comprising a headset with a display, wherein the virtual surgical environment is output on the display.

19. The system according to claim 15 , wherein the size or location of the workspace is further determined based on a procedure type.

20. The system according to claim 15 , wherein the virtual end effector includes a virtual trocar, a virtual stapler, a virtual grasper, or a virtual scissor.