Virtual reality surgical training systems
Disclosed herein are systems, methods, and software for providing a virtual environment with enhanced visual textures and haptic detail. In some embodiments, a texture atlas and UV mapping is used to render virtual objects having multiple textures that can be manipulated in real time. In some cases, UV coordinates are used to provide enhanced haptic detail.
1. A method for providing a virtual reality or augmented reality surgical simulation comprising a virtual object comprising a plurality of voxels, the method comprising:
providing a simulation platform operably associated with a display device, the simulation platform comprising a processor, a handheld component operatively connected to the processor, and a non-transitory computer readable storage medium coupled to the processor and encoded with a computer program;
identifying, via the simulation platform, a plurality of UV coordinates associated with the plurality of voxels of the virtual object, wherein the plurality of voxels are associated with tissue type data used to generate the plurality of UV coordinates;
locating, via the simulation platform, data comprising haptic information associated with the plurality of voxels, and a plurality of textures corresponding to the tissue type data associated with the plurality of voxels of the virtual object within a texture atlas, wherein each of the plurality of textures is located within the texture atlas using an offset value that is based on at least one UV coordinate associated with at least one voxel of the plurality of voxels;
mapping, via the simulation platform, the data comprising the plurality of textures onto the plurality of voxels of the virtual object;
displaying, via the simulation platform, the virtual object comprising the plurality of textures mapped onto the plurality of voxels within the surgical simulation;
providing, via the simulation platform, haptic feedback to a user through the handheld component, wherein the haptic feedback corresponds, in part, to bump map data corresponding to a surface of the virtual object; and
displaying, via the simulation platform, when a force is transmitted to the virtual object through the hand-held controller, a response of a texture associated with the virtual object to the force.
2. The method of claim 1 , wherein the hand-held component comprises a wand, a joystick, a mouse, a roller, a grasper, or a glove, wherein the hand-held component controls a virtual surgical instrument within the virtual or augmented reality surgical simulation, and wherein the virtual surgical instrument comprises a scalpel, a needle driver, a clamp, a clip applier, a surgical stapler, a retractor, a periosteal elevator, a rongeur, a nerve hook, a curette, an awl, a probe, a sagittal saw, a drill, a suture, a hammer, a finger, a laparoscopic instrument, an electrocautery, a suctioning instrument, or any combination thereof.
3. The method of claim 2 , further comprising displaying, via the simulation platform, a movement of the virtual surgical instrument in the surgical field in the same direction as a movement of the hand-held component based on the input.
4. The method of claim 1 , wherein the virtual reality or augmented reality surgical simulation comprises a representation of at least one of a bone, a muscle, an organ, a blood vessel, blood, and a nerve.
5. The method of claim 1 , wherein all textures that are mapped within the surgical simulation are positioned on the texture atlas and each mapped texture is associated with a set of unique UV coordinates.
6. The method of claim 1 , wherein the method further comprises displaying, via the simulation platform, a seamless movement of a texture within the virtual reality or augmented reality surgical simulation.
7. The method of claim 1 , wherein the method further comprises displaying, via the simulation platform, both an exterior texture and an interior texture of the virtual object when that virtual object is cut or altered to expose the interior texture with a virtual surgical instrument within the surgical simulation, and wherein the exterior texture and the interior texture are positioned on the texture atlas and each have a unique location within the texture atlas associated with unique UV coordinates.
8. The method of claim 7 , wherein the method further comprises simultaneously displaying the exterior texture and the interior texture and wherein the processor draws upon the texture atlas once.
9. The method of claim 1 , wherein the haptic feedback comprises a sensation that represents the response of the texture to the force.
10. The method of claim 1 , wherein the virtual object is rendered for display as a three-dimensional mesh generated from the plurality of voxels.
11. The method of claim 10 , wherein the plurality of textures is mapped onto the three-dimensional mesh using a triplanar shader.
12. The method of claim 10 , wherein the three-dimensional mesh is generated using an adaptive mesh algorithm.
13. The method of claim 12 , wherein the adaptive mesh algorithm is a marching cubes or tetrahedrons algorithm.
14. The method of claim 1 , wherein the surgical simulation comprises two or more virtual objects.
15. The method of claim 1 , wherein the virtual object is a simulated patient or an anatomic portion of the simulated patient.
16. The method of claim 15 , wherein the anatomic portion of the simulated patient comprises an arm, a leg, a torso, a head, or neck.
17. The method of claim 1 , wherein the display device comprises a head mounted device.
18. The method of claim 17 , wherein in the head mounted device comprises one or more of a virtual reality headset, a helmet, goggles, and eyeglasses.