Simulation features combining mixed reality and modular tracking
Techniques and systems are described for providing modularity, tracking and simulation features of a mixed reality simulator. A modular approach to tracking systems, tracked instruments, and interchangeable modular physical models is described. Enhanced indicators and indicator interfaces improve imaging probe and instrument orientation and alignment with respect to a physical target, including but not necessarily within a combined physical and virtual system; these include anisotropy indicators, indicators of alignment in both in-plane and out-of-plane ultrasound or other imaging techniques, and perpendicularity indicators for better placement of probes and instruments over curved surfaces. A universal needle hub is described that integrates tracking system components and simulation feedback components usably with varying needle types.
1. An instrument connector device comprising:
an instrument connecting component for removably connecting a first medical instrument to the instrument connector device; and
a second instrument connecting component for connecting at least a second medical instrument to the instrument connector device, wherein the second instrument comprises an imaging device, wherein the instrument connector device is configured to provide at least one of haptic feedback, tactile feedback, auditory feedback, or visual feedback via a feedback unit, in response to an interaction in a mixed reality system comprising at least one of a (a) virtual model or (b) a physical model, wherein the at least one of the haptic feedback, the tactile feedback, the auditory feedback, or the visual feedback reflects the interaction of the first medical instrument relative to an imaging plane of the imaging device.
2. The instrument connector device according to claim 1 , wherein the first medical instrument is a needle, and the needle is connectable to the instrument connecting component by at least needle tubing.
3. The instrument connector device according to claim 1 , wherein the instrument connector device is communicatively connectable to at least one sensor configured to track at least one of a position or an orientation of the first medical instrument, wherein the haptic, tactile, auditory, or visual feedback is dependent upon at least one of the tracked position or the orientation of the first medical instrument.
4. The instrument connector device according to claim 1 , wherein when a syringe comprising a syringe plunger is connected to the instrument connector device via the second instrument connecting component, a resistance to movement of the syringe plunger varies responsive to at least one of a pushing of or pulling on the syringe plunger.
5. The instrument connector device according to claim 4 , wherein the feedback unit comprises at least one valve or actuator configured to regulate varying pressures in a space between a syringe tip and a plunger face of the syringe.
6. The instrument connector device according to claim 1 , wherein the first medical instrument is a needle connected to the instrument connector device via the instrument connecting component, wherein the second medical instrument is a syringe comprising a syringe plunger and is connected to the instrument connector device via the second instrument connecting component, and wherein resistance to movement of the syringe plunger varies dependent upon at least one of a position or an orientation of a tip of the needle relative to a virtual object of the virtual model, or a physical object of the physical model.
7. The instrument connector device according to claim 6 , wherein at least one of the physical object or the virtual object is or is associated with a vessel, cavity, organ, location, or potential space.
8. The instrument connector device according to claim 1 , connectable to or further comprising a vibration motor configured to simulate different tactile qualities depending on a type of tissue associated with a portion of the virtual model or the physical model with which the first medical instrument interacts.
9. The instrument connector device according to claim 1 , further comprising at least one magnetic tracking sensor, wherein at least one of the first medical instrument or the second medical instrument comprises non-magnetic materials.
10. The instrument connector device according to claim 1 , wherein the first medical instrument comprises at least one of a needle for medical procedures, a syringe, an introducer, a dilator, a catheter, tubes, a guidewire, a biopsy device, a needle ablation device, an electrocautery needle, a brachytherapy instrument, a cryotherapy instrument, a thermotherapy instrument, a drainage device, or a fine needle aspiration device.
11. A feedback system comprising:
a tracking system configured to track at least one of a position or an orientation of at least one medical instrument with respect to at least two interchangeable physical models, wherein the tracking system is at least one of: (a) attachable to at least one stand configured to interchangeably receive any of the at least two interchangeable physical models, or (b) movable to be positioned approximate to any of the at least two interchangeable physical models; and
one or more processors configured to cause display of at least one of a virtual representation, or a three dimensional (3D) visualization, to further reflect a visual representation of any of the at least two interchangeable physical models attached to or approximate the tracking system, and at least one of a position or an orientation of the at least one medical instrument relative to any of the at least two interchangeable physical models attached to or approximate the tracking system, wherein the one or more processors are further configured to:
supplement at least one of the virtual representation or the 3D visualization with visual feedback reflecting an angle of an imaging plane relative to at least one of (a) a physical model, a virtual model, or a portion thereof, or (b) another instrument.
12. The feedback system according to claim 11 , wherein the tracking system comprises a tracker box device encapsulating an interface to a transmitter device, wherein the tracker box device protects hardware connectors within the tracker box device, wherein the at least one medical instrument is connectable to the tracker box device.
13. The feedback system according to claim 11 , wherein the virtual representation is a simulated ultrasound image.
14. An apparatus comprising on or more processors and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the one or more processors, cause the apparatus to at least:
receive an indication of at least one of a position or an orientation of a tracked instrument relative to a physical model or a virtual model, the tracked instrument having an associated imaging plane;
cause display of at least one of a virtual representation or a 3D visualization of the physical model or the virtual model, wherein in an instance an angle of the imaging plane relative to at least one of (a) the physical model or the virtual model or portion thereof, or (b) another instrument, has a first predefined relationship with a predetermined threshold angle, cause the displayed virtual representation or the displayed 3D visualization to have a first display property, and in an instance the angle of the imaging plane relative to at least one of (a) the physical model, the virtual model, or a portion thereof, or (b) another instrument, has a second predefined relationship with the predetermined threshold angle, cause the displayed virtual representation or the displayed 3D visualization to have a second display property.
15. The apparatus according to claim 14 , wherein the first predefined relationship comprises the angle being equal to or less than the predetermined threshold angle, and the second predefined relationship comprises the angle being greater than the predetermined threshold angle.
16. The apparatus according to claim 14 , wherein the first predefined relationship comprises the angle being less than the predetermined threshold angle, and the second predefined relationship comprises the angle being equal to or greater than the predetermined threshold angle.
17. The apparatus according to claim 14 , wherein causing the displayed virtual representation or the displayed 3D visualization to have the second display property comprises causing at least one of an imaged instrument, an imaged organ, an imaged tissue, or at least a portion of the displayed virtual representation or the displayed 3D visualization to change in brightness or disappear to simulate a change in echogenicity.
18. The apparatus according to claim 14 , wherein in an instance the angle has the second predefined relationship with the predetermined threshold angle, the at least one memory and the computer program code are configured to, with the one or more processors, cause the apparatus to cause a notification to be provided.
19. The apparatus according to claim 14 , wherein the at least one memory and the computer program code configured to, with the one or more processors, cause the apparatus to cause display of a clock-face indicator indicating a rotation of the tracked instrument relative to an anatomical plane.
20. The apparatus according to claim 14 , wherein the at least one memory and the computer program code configured to, with the one or more processors, cause the apparatus to at least:
receive an indication of an input indicative of hydrolocation or hydro-dissection; and
modify at least one of the virtual representation or the 3D visualization to show an effect of a pool of liquid delivered via a needle based upon a location of a tracked needle tip.