System including a non-transitory computer readable program storage unit encoded with instructions that, when executed by a computer, perform a method for three-dimensional augmented reality guidance for use of medical equipment
A medical guidance system providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a novice user of medical equipment having limited medical training, to achieve improved diagnostic or treatment outcomes.
1. A system, comprising:
an augmented reality user interface (ARUI);
a medical equipment system;
a library configured to store non-transitory computer readable data; and
a non-transitory computer readable program storage unit encoded with instructions that, when executed by a computer, perform a method for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a user of a medical equipment system, the method comprising:
receiving data from the medical equipment system during a medical procedure performed by a user of the medical equipment to achieve a medical procedure outcome;
receiving real-time user positioning data relating to one or more of the movement, position, and orientation of at least a portion of the medical equipment system within a volume of the user's environment during the medical procedure performed by the user;
retrieving from the library at least one of 1) stored reference positioning data relating to one or more of the movement, position, and orientation of at least a portion of the medical equipment system during reference a medical procedure, and 2) stored reference outcome data relating to a reference performance of the medical procedure;
comparing at least one of 1) the sensed real-time user positioning data to the retrieved reference positioning data, and 2) the data received from the medical equipment system during a medical procedure performed by the user to the retrieved reference outcome data;
generating at least one of 1) real-time position-based 3D AR feedback based on the comparison of the sensed real-time user positioning data to the retrieved reference positioning data, and 2) real-time output-based 3D AR feedback based on the comparison of the data received from the medical equipment system during a medical procedure performed by the user to the retrieved reference outcome data; and
providing at least one of the real-time position-based 3D AR feedback and the real-time output-based 3D AR feedback to the user via the ARUI.
2. The system of claim 1 , wherein the medical procedure performed by a user of the medical equipment comprises a first medical procedure, and the stored reference positioning data and stored reference outcome data relate to a reference performance of the first medical procedure prior to the user's performance of the first medical procedure.
3. The system of claim 1 , wherein the medical procedure performed by a user of the medical equipment comprises a first ultrasound procedure, and the stored reference positioning data and stored reference outcome data comprise ultrasound images obtained during a reference performance of the first ultrasound procedure prior to the user's performance of the first ultrasound procedure.
4. The system of claim 3 , wherein sensing real-time user positioning data comprises sensing real-time movement by the user of an ultrasound probe relative to the body of a patient.
5. The system of claim 1 , wherein the non-transitory computer readable program storage unit further comprises a neural network, and the method further comprises generating real-time outcome-based 3D AR feedback based on a comparison, using the neural network, of real-time images generated by the user in an ultrasound procedure to retrieved images generated during a reference performance of the same ultrasound procedure prior to the user.
6. The system of claim 5 , wherein the neural network is a convolutional neural network.
7. The system of claim 1 , further comprising a sensor comprising at least one of a magnetic GPS system, a digital camera tracking system, an infrared camera system, an accelerometer, and a gyroscope, wherein the sensor provides, to the non-transitory computer readable program storage unit, the real-time user positioning data.
8. The system of claim 7 , wherein said sensor is configured to sense at least one of:
a magnetic field generated by said at least a portion of the medical equipment system;
the movement of one or more passive visual markers coupled to one or more of the patient, a hand of the user, or a portion of the medical equipment system; and
the movement of one or more active visual markers coupled to one or more of the patient, a hand of the user, or a portion of the medical equipment system.
9. The system of claim 1 , wherein providing at least one of the real-time position-based 3D AR feedback and the real-time output-based 3D AR feedback to the user comprises providing a feedback selected from:
a virtual prompt indicating a movement correction to be performed by a user;
a virtual image or video instructing the user to change the orientation of a probe to match a desired orientation;
a virtual image or video of a correct motion path to be taken by the user in performing a medical procedure;
a color-coded image or video indicating correct and incorrect portions of the user's motion in performing a medical procedure;
an instruction to a user to press an ultrasound probe deeper or shallower into tissue to focus the ultrasound image on a desired target structure of the patient's body;
an auditory instruction, virtual image, or virtual video indicating a direction for the user to move an ultrasound probe; and
tactile information.
10. The system of claim 1 , wherein providing at least one of the real-time position-based 3D AR feedback and the real-time output-based 3D AR feedback comprises providing both of the real-time position-based 3D AR feedback and the real-time output-based 3D AR feedback to the user.
11. The system of claim 1 , wherein said ARUI is configured to provide said at least one of the real-time position-based 3D AR feedback and the real-time output-based 3D AR feedback to a head mounted display (HMD) worn by the user.
12. A non-transitory computer readable program storage unit encoded with instructions that, when executed by a computer, perform a method for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a user of a medical equipment system, the method comprising:
receiving, from a medical equipment system, data during a medical procedure performed by a user of the medical equipment to achieve a medical procedure outcome;
receiving, from a sensor, real-time user positioning data relating to one or more of the movement, position, and orientation of at least a portion of the medical equipment system within a volume of the user's environment during the medical procedure performed by the user;
retrieving, from a library, at least one of 1) stored reference positioning data relating to one or more of the movement, position, and orientation of at least a portion of the medical equipment system during reference a medical procedure, and 2) stored reference outcome data relating to a reference performance of the medical procedure;
comparing at least one of 1) the sensed real-time user positioning data to the retrieved reference positioning data, and 2) the data received from the medical equipment system during a medical procedure performed by the user to the retrieved reference outcome data;
generating 1) real-time position-based 3D AR feedback based on the comparison of the sensed real-time user positioning data to the retrieved reference positioning data, and 2) real-time output-based 3D AR feedback based on the comparison of the data received from the medical equipment system during a medical procedure performed by the user to the retrieved reference outcome data; and
providing, via an augmented reality user interface (ARUI), at least one of the real-time position-based 3D AR feedback and the real-time output-based 3D AR feedback to the user.
13. The non-transitory computer readable program storage unit of claim 12 , wherein the medical procedure performed by a user of the medical equipment comprises a first medical procedure, and the stored reference positioning data and stored reference outcome data relate to a reference performance of the first medical procedure prior to the user's performance of the first medical procedure.
14. The non-transitory computer readable program storage unit of claim 12 , wherein the non-transitory computer readable program storage unit further comprises a neural network, and the method further comprises generating real-time outcome-based 3D AR feedback based on a comparison, using the neural network, of real-time images generated by the user in an ultrasound procedure to retrieved images generated during a reference performance of the same ultrasound procedure prior to the user.
15. The non-transitory computer readable program storage unit of claim 14 , wherein the neural network is a convolutional neural network.
16. The non-transitory computer readable program storage unit of claim 12 , wherein said sensor comprises at least one of a magnetic GPS system, a digital camera tracking system, an infrared camera system, an accelerometer, and a gyroscope, wherein the sensor provides, to the non-transitory computer readable program storage unit, the real-time user positioning data.
17. The non-transitory computer readable program storage unit of claim 16 , wherein said sensor is configured to sense at least one of:
a magnetic field generated by said at least a portion of the medical equipment system;
the movement of one or more passive visual markers coupled to one or more of the patient, a hand of the user, or a portion of the medical equipment system; and
the movement of one or more active visual markers coupled to one or more of the patient, a hand of the user, or a portion of the medical equipment system.
18. The non-transitory computer readable program storage unit of claim 12 , wherein providing at least one of the real-time position-based 3D AR feedback and the real-time output-based 3D AR feedback to the user comprises providing a feedback selected from:
a virtual prompt indicating a movement correction to be performed by a user;
a virtual image or video instructing the user to change the orientation of a probe to match a desired orientation;
a virtual image or video of a correct motion path to be taken by the user in performing a medical procedure;
a color-coded image or video indicating correct and incorrect portions of the user's motion in performing a medical procedure;
an instruction to a user to press an ultrasound probe deeper or shallower into tissue to focus the ultrasound image on a desired target structure of the patient's body;
an auditory instruction, virtual image, or virtual video indicating a direction for the user to move an ultrasound probe; and
tactile information.
19. The non-transitory computer readable program storage unit of claim 12 , wherein said ARUI is configured to provide said at least one of the real-time position-based 3D AR feedback and the real-time output-based 3D AR feedback to a head mounted display (HMD) worn by the user.
20. A medical guidance system for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance in the use of a medical equipment system, the medical guidance system comprising:
(I) a computer comprising:
(A) a medical equipment interface to a medical equipment system, wherein said medical equipment interface receives data from the medical equipment system during a medical procedure performed by a user to achieve a medical procedure outcome;
(B) an augmented reality user interface (ARUI) to an AR head mounted display (HMD) for presenting information pertaining to both real and virtual objects to the user during the performance of the medical procedure;
(C) a guidance system interface (GSI) to a three-dimensional guidance system (3DGS) that senses real-time user positioning data relating to one or more of the movement, position, and orientation of at least a portion of the medical equipment system within a volume of a user's environment during a medical procedure performed by the user;
(D) a library containing 1) stored reference positioning data relating to one or more of the movement, position, and orientation of at least a portion of the medical equipment system during a reference medical procedure and 2) stored reference outcome data relating to an outcome of a reference performance of the reference medical procedure; and
(E) a machine learning module (MLM) for providing at least one of 1) position-based 3D AR feedback to the user based on the sensed user positioning data and 2) outcome-based 3D AR feedback to the user based on the medical procedure outcome, the MLM comprising:
(i) a position-based feedback module comprising
(a) a first module for receiving and analyzing real-time user positioning data;
(b) a second module for comparing the user positioning data to the stored reference positioning data, and
(c) a third module for generating real-time position-based 3D AR feedback based on the output of the second module, and providing said real-time position-based 3D AR feedback to the user via the ARUI and the AR HMD; and
(ii) an outcome-based feedback module comprising
(a) a fourth module for receiving real-time data from the medical equipment system via said medical equipment interface as the user performs the medical procedure;
(b) a fifth module for comparing the real-time data received from the medical equipment system as the user performs the medical procedure to the stored reference outcome data, and
(c) a sixth module for generating real-time outcome-based 3D AR feedback based on the output of the fifth module and providing said real-time outcome-based 3D AR feedback to the user via the ARUI and the AR HMD.