Virtual reality surgical training systems with advanced haptic feedback
Disclosed herein are systems, methods, and software for providing a virtual environment with enhanced visual and haptic detail. In some embodiments, the haptic tool and haptic target are assigned affordance and susceptibility values, respectively, that are used to determine visual and haptic feedback.
1. A surgical training system, comprising
(a) a processor;
(b) a hand-held component operatively coupled to the processor and configured to provide haptic feedback to a user;
(c) a non-transitory computer readable storage medium encoded with a computer program that causes the processor to:
i. display a virtual object as part of a virtual surgical field, wherein the virtual object is associated with a damage threshold value, a range of units of pressure having a lower bound and an upper bound, and a plurality of susceptibility values corresponding to a plurality of interactions, wherein the susceptibility values define a susceptibility of the virtual object to damage by an interaction with a virtual surgical instrument, and the damage threshold value is an abstraction of a minimum amount of force or pressure necessary to cause an effect on the virtual object and/or result in a type and/or amount of haptic feedback;
ii. display a movement of a first virtual surgical instrument within the virtual surgical field based on a first input from the hand-held component when a user moves the hand-held component, wherein the first virtual surgical instrument is associated with a first plurality of affordance values corresponding to the plurality of interactions, wherein the affordance values define a strength of interaction by the first virtual surgical instrument with the virtual object, wherein the first virtual surgical instrument and the virtual object engage in a first interaction selected from the plurality of interactions;
iii. cross-reference the plurality of susceptibility values of the virtual object with the plurality of affordance values of the first virtual surgical instrument to calculate a first damage multiplier value based on a first susceptibility value of the plurality of susceptibility values and a first affordance value of the plurality of affordance values each corresponding to the first interaction and algorithmically calculate a first haptic value corresponding to the first interaction when the first virtual surgical instrument is displayed in the first interaction with the virtual object within the virtual surgical field, wherein calculation of the first haptic value is based on the damage threshold value, the first damage multiplier, the first susceptibility value, the first input, and the first affordance value;
iv. transmit the first haptic value as a first haptic feedback to the hand-held component wherein a quantity of the first haptic feedback that is transmitted is algorithmically calculated based on the first haptic value;
v. display a movement of a second virtual surgical instrument within the virtual surgical field based on a second input from the hand-held component when a user moves the hand-held component, wherein the second virtual surgical instrument is associated with a second plurality of affordance values corresponding to the plurality of interactions, wherein the affordance values define a strength of interaction by the second virtual surgical instrument with the virtual object, wherein the second virtual surgical instrument and the virtual object engage in a second interaction selected from the plurality of interactions;
vi. cross-reference the plurality of susceptibility values of the virtual object with the plurality of affordance values of the second virtual surgical instrument to calculate a second damage multiplier value based on a second susceptibility value of the plurality of susceptibility values and a second affordance value of the plurality of affordance values each corresponding to the second interaction and algorithmically calculate a second haptic value corresponding to the second interaction when the second virtual surgical instrument is displayed in the second interaction with the virtual object within the virtual surgical field, wherein calculation of the second haptic value is based on the damage threshold value, the second damage multiplier, the second susceptibility value, the second input, and the second affordance value; and
vii. transmit the second haptic value as a second haptic feedback to the hand-held component wherein a quantity of the second haptic feedback that is transmitted is algorithmically calculated based on the second haptic value, wherein the damage threshold value is divided by the first damage multiplier and/or the second damage multiplier to calculate the units of pressure a user must apply to achieve a desired effect on the virtual object, wherein, for a given interaction between the virtual object and the first and second virtual surgical instrument, algorithmic calculation of the haptic feedback generated by the given interaction is achieved via an abstraction layer such that a resultant effect of the given interaction is not pre-determined.
2. The system of claim 1 , wherein the first virtual surgical instrument comprises a surgical instrument selected from a scalpel, a needle driver, a clamp, a clip applier, a surgical stapler, a retractor, a periosteal elevator, a rongeur, a nerve hook, a curette, an awl, a probe, a sagittal saw, a drill, a suture, a hammer, a finger, a laparoscopic instrument, an electrocautery, or a suctioning instrument;
the second virtual surgical instrument comprises a surgical instrument selected from a scalpel, a needle driver, a clamp, a clip applier, a surgical stapler, a retractor, a periosteal elevator, a rongeur, a nerve hook, a curette, an awl, a probe, a sagittal saw, a drill, a suture, a hammer, a finger, a laparoscopic instrument, an electrocautery, or a suctioning instrument; and wherein the first virtual surgical instrument and the second virtual surgical instrument are different.
3. The system of claim 1 , wherein the first interaction is selected from the list consisting of cutting, sawing, drilling, injecting, stapling, clipping, clamping, driving, retracting, elevating, curetting, cauterizing, sucking, hammering, and stitching; and
the second interaction is selected from the list consisting of cutting, sawing, drilling, injecting, stapling, clipping, clamping, driving, retracting, elevating, curetting, cauterizing, sucking, hammering, and stitching; and
wherein the first interaction and the second interaction are different.
4. The system of claim 1 , wherein the virtual object comprises a representation of at least one of a bone, a muscle, an organ, a blood vessel, blood, and a nerve.
5. The system of claim 1 , wherein the virtual object comprises a representation of at least one tissue.
6. The system of claim 1 , further comprising a second virtual object.
7. The system of claim 6 , wherein the virtual object and the second virtual object comprise a different type of tissue.
8. The system of claim 6 , wherein the virtual object and the second virtual object comprise the same type of tissue.
9. The system of claim 1 , further comprising wherein the computer program causes the processor to display a first visual effect corresponding to the first interaction and a second visual effect corresponding to the second interaction.
10. The system of claim 9 , wherein the first visual effect comprises a deformation, a burning, a color change or discoloration, a clean cut, a minor cut, tearing or ripping, grinding, freezing, or any combination thereof;
wherein the second visual effect comprises a deformation, a burning, a color change or discoloration, a clean cut, a minor cut, tearing or ripping, grinding, freezing, or any combination thereof; and
the first visual effect is different than the second visual effect.
11. The system of claim 1 , wherein the first input comprises a first force applied by the user to the hand-held component, wherein the first force corresponds to at least one of a gripping force, a pushing force, a pulling force, or a rotational force;
the second input comprises a second force applied by the user to the hand-held component, wherein the second force corresponds to at least one of a gripping force, a pushing force, a pulling force, or a rotational force; and
wherein the first input is different from the second input.
12. The system of claim 1 , wherein the hand-held component comprises a wand, a joystick, a mouse, a roller, a grasper, a glove, an arm, a keyboard, a pointer, a wand, or a handle.
13. The system of claim 12 , wherein the hand-held component comprises the glove; and
wherein the glove comprises a haptic glove.
14. A computer implemented method comprising:
(a) displaying a virtual object as part of a virtual surgical field, wherein the virtual object is associated with a damage threshold value, a range of units of pressure having a lower bound and an upper bound, and a plurality of susceptibility values corresponding to a plurality of interactions, wherein the susceptibility values define a susceptibility of the virtual object to damage by an interaction with a virtual surgical instrument, and the damage threshold value is an abstraction of a minimum amount of force or pressure necessary to cause an effect on the virtual object and/or result in a type and/or amount of haptic feedback;
(b) displaying a movement of a first virtual surgical instrument within the virtual surgical field based on a first input from a hand-held component when a user moves the hand-held component, wherein the first virtual surgical instrument is associated with a first plurality of affordance values corresponding to the plurality of interactions, wherein the affordance values define a strength of interaction by the first virtual surgical instrument with the virtual object, wherein the first virtual surgical instrument and the virtual object engage in a first interaction selected from the plurality of interactions;
(c) cross-referencing the plurality of susceptibility values of the virtual object with the plurality of affordance values of the first virtual surgical instrument to calculate a first damage multiplier value based on a first susceptibility value of the plurality of susceptibility values and a first affordance value of the plurality of affordance values each corresponding to the first interaction, and calculating, via one or more algorithms, a first haptic value corresponding to the first interaction when the first virtual surgical instrument is displayed in the first interaction with the virtual object within the virtual surgical field, wherein calculation of the first haptic value is based on the damage threshold value, the first damage multiplier, the first susceptibility value, the first input, and the first affordance value;
(d) transmitting a first haptic value as a first haptic feedback to the hand-held component wherein a quantity of the first haptic feedback that is transmitted is algorithmically calculated based on the first haptic value;
(e) displaying a movement of a second virtual surgical instrument within the virtual surgical field based on a second input from a hand-held component when a user moves the hand-held component, wherein the second virtual surgical instrument is associated with a second plurality of affordance values corresponding to the plurality of interactions, wherein the affordance values define a strength of interaction by the second virtual surgical instrument with the virtual object, wherein the second virtual surgical instrument and the virtual object engage in a second interaction selected from the plurality of interactions;
(f) cross-referencing the plurality of susceptibility values of the virtual object with the plurality of affordance values of the second virtual surgical instrument to calculate a second damage multiplier value based on a second susceptibility value of the plurality of susceptibility values and a second affordance value of the plurality of affordance values each corresponding to the second interaction and calculating, via one or more algorithms a second haptic value corresponding to the second interaction when the second virtual surgical instrument is displayed in the second interaction with the virtual object within the virtual surgical field, wherein calculation of the second haptic value is based on the damage threshold value, the second damage multiplier, the second susceptibility value, the second input, and the second affordance value; and
(g) transmitting a second haptic feedback to the hand-held component wherein a quantity of the second haptic feedback that is transmitted is calculated based on the second haptic value, wherein the damage threshold value is divided by the first damage multiplier and/or the second damage multiplier to calculate the units of pressure a user must apply to achieve a desired effect on the virtual object, wherein, for a given interaction between the virtual object and the first and second virtual surgical instrument, algorithmic calculation of the haptic feedback generated by the given interaction is achieved via an abstraction layer such that a resultant effect of the given interaction is not pre-determined.
15. The method of claim 14 , wherein the first virtual surgical instrument comprises a surgical instrument selected from a scalpel, a needle driver, a clamp, a clip applier, a surgical stapler, a retractor, a periosteal elevator, a rongeur, a nerve hook, a curette, an awl, a probe, a sagittal saw, a drill, a suture, a hammer, a finger, a laparoscopic instrument, an electrocautery, or a suctioning instrument;
the second virtual surgical instrument comprises a surgical instrument selected from a scalpel, a needle driver, a clamp, a clip applier, a surgical stapler, a retractor, a periosteal elevator, a rongeur, a nerve hook, a curette, an awl, a probe, a sagittal saw, a drill, a suture, a hammer, a finger, a laparoscopic instrument, an electrocautery, or a suctioning instrument; and wherein the first virtual surgical instrument and the second virtual surgical instrument are different.
16. The method of claim 14 , wherein the first interaction is selected from the list consisting of cutting, sawing, drilling, injecting, stapling, clipping, clamping, driving, retracting, elevating, curetting, cauterizing, sucking, hammering, and stitching; and
the second interaction is selected from the list consisting of cutting, sawing, drilling, injecting, stapling, clipping, clamping, driving, retracting, elevating, curetting, cauterizing, sucking, hammering, and stitching; and
wherein the first interaction and the second interaction are different.
17. The method of claim 14 , wherein the virtual object comprises a representation of at least one of a bone, a muscle, an organ, a blood vessel, blood, and a nerve.
18. The method of claim 14 , further comprising a second virtual object.
19. The method of claim 18 , wherein the second virtual object comprises a representation of at least one tissue.
20. The method of claim 14 , wherein the first input comprises a first force applied by the user to the hand-held component, wherein the first force corresponds to at least one of a gripping force, a pushing force, a pulling force, or a rotational force;
the second input comprises a second force applied by the user to the hand-held component, wherein the second force corresponds to at least one of a gripping force, a pushing force, a pulling force, or a rotational force;
and wherein the first input is different from the second input.
21. The system of claim 1 , wherein calculating the first damage multiplier comprises computationally integrating and/or transforming the plurality of susceptibility values of the virtual object and the plurality of affordance values of the first virtual surgical instrument to generate the first damage multiplier, wherein the first damage multiplier represents a degree and/or rate at which one or more of a movement, a pressure, and/or force applied by the first surgical instrument upon the virtual object in the first interaction, in association with the user movement of the hand-held component, results in damage to the virtual object, and/or the type and/or amount of the first haptic feedback associated with the first interaction.
22. The system of claim 1 , wherein calculating the second damage multiplier comprises computationally integrating and/or transforming the plurality of susceptibility values of the virtual object and the plurality of affordance values of the second virtual surgical instrument to generate the second damage multiplier, wherein the second damage multiplier represents a degree and/or rate at which one or more of a movement, a pressure, and/or force applied by the second surgical instrument upon the virtual object in the second interaction, in association with the user movement of the hand-held component, results in damage to the virtual object, and/or the type and/or amount of second haptic feedback associated with the second interaction.
23. The method of claim 14 , wherein calculating the first damage multiplier comprises computationally integrating and/or transforming the plurality of susceptibility values of the virtual object and the plurality of affordance values of the first virtual surgical instrument to generate the first damage multiplier, wherein the first damage multiplier represents a degree and/or rate at which one or more of a movement, a pressure, and/or force applied by the first surgical instrument upon the virtual object in the first interaction, in association with the user movement of the hand-held component, results in damage to the virtual object, and/or the type and/or amount of the first haptic feedback associated with the first interaction.
24. The method of claim 14 , wherein calculating the second damage multiplier comprises computationally integrating and/or transforming the plurality of susceptibility values of the virtual object and the plurality of affordance values of the second virtual surgical instrument to generate the second damage multiplier, wherein the second damage multiplier represents a degree and/or rate at which one or more of a movement, a pressure, and/or force applied by the second surgical instrument upon the virtual object in the second interaction, in association with the user movement of the hand-held component, results in damage to the virtual object, and/or the type and/or amount of second haptic feedback associated with the second interaction.