Surgical navigation inside a body
A virtual reality surgical navigation method includes the steps of preparing a multi dimension virtual model associated with an anatomy inside of patient; receiving data indicative of a surgeon's current head position, including direction of view and angle of view; rendering a first virtual three-dimensional image from the virtual model, the virtual three-dimensional image being representative of an anatomical view from a first perspective at a location inside the patient, wherein the perspective is determined by data indicative of the surgeon's current head position; communicating the first rendered virtual image to a virtual headset display; receiving data input indicative of the surgeon's head moving to a second position, wherein the head movement comprises at least one of a change in angle of view and a change in direction of view; and rendering a second virtual three-dimensional image from the virtual model, the second virtual three-dimensional image being representative of an anatomical view from a second perspective at a first location inside the patient.
1. An augmented reality surgical navigation system comprising:
one or more processors;
one or more computer-readable tangible storage devices;
at least one sensor for detecting information about a user's position and motion around a patient;
at least one camera for receiving live images of internal anatomical features of the patient; and
program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors, said program instructions comprising:
first program instructions for preparing a multi dimension virtual model of the internal anatomical features of the patient, said model configured for providing the user with dynamic interaction of the internal anatomical features provided in the model;
second program instructions for receiving tracking information indicative of a user's current view of the patient, including the user's position and motion around the patient as detected by the sensor and the user's angle of view of the patient;
third program instructions for identifying in the virtual model a virtual view based on the received tracking information, wherein the identified virtual view corresponds to the user's view of the patient;
fourth program instructions for rendering a virtual image from the virtual model based on the identified virtual view, said virtual view showing dynamic interactions of the user with the internal anatomical features of the model based on the user's current view of the patient according to the user's position and motion around the patient permitting user dynamic interaction with the internal anatomical features of the model in three dimensions such that said user can see and interact with the virtual image over 360 degrees around the image including the front and back of the image; and
fifth program instructions for communicating the rendered virtual image to a display where the rendered virtual image is combined with the live images of the internal anatomical features of the patient and the user's view to form an augmented reality view of the patient.
2. The system of claim 1 , wherein:
the second program instructions are further configured to continuously, in real time, receive updated tracking information as a user's position relative to the patient and the user's angle of view of the patient changes; and
the third program instructions are further configured to continuously, in real time, identify virtual views corresponding to the continuously updated tracking information such that the virtual views are continuously synchronized with the user's view of the patient.
3. The system of claim 1 , wherein the program instructions further comprise:
sixth program instructions for receiving a second live video feed corresponding to the user's view of the patient; and
seventh program instructions for generating an augmented reality view of the patient by continuously rendering in real time and combining the virtual image with the second live video feed, wherein the virtual image is synchronized with the second live video feed based on the tracking information.
4. The system of claim 1 , wherein the first program instructions for preparing the virtual model is configured to build the model based on received patient specific DICOM data.
5. The system of claim 1 , wherein the program instructions further include eighth program instructions for receiving an additional data input, and wherein the third program instructions are configured to integrate the additional data input with the rendered virtual image.
6. The system of claim 1 , wherein:
the second program instructions are further configured to receive second tracking information indicative of a second user's current view of the patient, including the second user's position relative to the patient and the second user's angle of view of the patient, wherein the second user's current view is different than the user's current view of the patient;
the third program instructions are further configured to identify in the virtual model a second virtual view based on the received second tracking information, wherein the identified second virtual view corresponds to the second user's view of the patient;
the fourth program instructions are further configured to render a second virtual image from the virtual model based on the identified second virtual view; and
the fifth program instructions are further configured to communicate the rendered second virtual image to a second display where the rendered second virtual image is combined with the second user's view to form a second augmented reality view of the patient, simultaneous to communicating the first rendered virtual image to the first display.
7. The system of claim 1 , wherein third program instructions are configured to receive tracking information from one of a plurality of sensors disposed throughout a room, thereby enabling tracking a location 360 degrees around the patient.
8. The system of claim 1 , wherein third program instructions are configured to receive tracking information from one or more sensors disposed on a HDM worn by a user, thereby enabling tracking a location 360 degrees around the patient.
9. The system of claim 1 , wherein said live images of the anatomical features of the patient are provided by an endoscope camera used on the patient to provide real-time patient organ data to support images in the augmented reality view.
10. The system of claim 9 , wherein an image of said endoscope camera is included in said augmented reality view.
11. The system of claim 1 , wherein said sensor is mounted on a head mounted display worn by the user, and wherein said sensor is configured to directly detect a location or motion of the user or a physical feature of a patient.
12. An augmented reality surgical navigation method, comprising the steps of:
preparing a multi dimension virtual model associated with a patient said model configured for providing a user with dynamic interaction of the internal anatomical features provided in the model;
capturing live images of internal anatomical features of the patient;
receiving tracking information based on directly detecting movement or location of the user indicative of a user's current view of the patient, including the user's position and motion around the patient and the user's angle of view of the patient;
identifying in the virtual model a virtual view based on the received tracking information, wherein the identified virtual view corresponds to the user's view of the patient;
rendering a virtual image from the virtual model based on the identified virtual view, said virtual view showing dynamic interactions of the user with the internal anatomical features of the model based on the user's current view of the patient according to the user's position and motion around the patient permitting user dynamic interaction with the internal anatomical features of the model in three such that said user can see and interact with the virtual image over 360 degrees around the image including the front and back of the image; and
communicating the rendered virtual image to a display where the rendered virtual image is combined with the live images of the internal anatomical features of the patient based on the user's view to form an augmented reality view of the patient that is displayed by said display.
13. The method of claim 12 , further comprising the steps of, in real time:
receive updated tracking information as a user's position relative to the patient and the user's angle of view of the patient changes; and
identify virtual views corresponding to the updated tracking information such that the virtual views are continuously synchronized with the live images of the internal anatomical features of the patient and the user's view of the patient.
14. The method of claim 12 , further comprising the step of
generating the augmented reality view of the patient by continuously rendering in real time and combining the virtual image with the live images of the internal anatomical features of the patient, wherein the virtual image is synchronized with the live images based on the tracking information.
15. The method of claim 12 , further comprising the steps of:
receiving second tracking information indicative of a second user's current view of the patient, including the second user's position relative to the patient and the second user's angle of view of the patient, wherein the second user's current view is different than the user's current view of the patient;
identifying in the virtual model a second virtual view based on the received second tracking information, wherein the identified second virtual view corresponds to the second user's view of the patient;
rendering a second virtual image from the virtual model based on the identified second virtual view; and
communicating the rendered second virtual image to a second display where the rendered second virtual image is combined with the second user's view to form a second augmented reality view of the patient, simultaneous to communicating the first rendered virtual image to the first display.
16. The method of claim 12 , further comprising the step of receiving tracking information about a location of the patient or a part of the patient, wherein said augmented reality view is rendered using information about the location of the patient.
17. The method of claim 12 , wherein said live images of the internal anatomical features of the patient are provided by an endoscope camera used on the patient to provide real-time patient organ data to support images in the augmented reality view.
18. The method of claim 17 , wherein an image of said endoscope camera is included in said augmented reality view.
19. The method of claim 17 , wherein said sensor is mounted on a head mounted display worn by the user, and wherein said sensor is configured to directly detect a location or motion of the user or a physical feature of a patient.
20. The method of claim 12 , further comprising the step of providing an HMD live video feed capturing video of the patient observed by the user of the HMD, wherein said augmented reality view of the patient also includes images from the HMD live video feed.
21. The method of claim 12 , further comprising the step of providing a tracking mechanism on a device that captures the live images of the internal anatomical feature, and wherein said tracking information includes tracking information from said tracking mechanism.
22. An augmented reality surgical navigation method, comprising the steps of:
preparing a multi dimension virtual model associated with a patient said model configured for providing the user with dynamic interaction of the internal anatomical features provided in the model;
receiving tracking information indicative of a user's current view of the patient, including the user's position and motion around the patient and the user's angle of view of the patient;
receiving tracking information about a location of the patient or a part of the patient;
capturing live images of internal anatomical features of the patient;
identifying in the virtual model a virtual view based on the received tracking information, wherein the identified virtual view corresponds to the user's view of the patient and the tracking information about the patient;
rendering a virtual image from the virtual model based on the identified virtual view, said virtual view showing dynamic interactions of the user with the internal anatomical features of the model based on the user's current view of the patient according to the user's position and motion around the patient permitting user dynamic interaction with the internal anatomical features of the model in three dimensions such that said user can see and interact with the virtual image over 360 degrees around the image including the front and back of the image; and
communicating the rendered virtual image to a display where the rendered virtual image is combined with the live images of internal anatomical features of the patient to form an augmented reality view of the patient that is displayed by said display.
23. The method of claim 22 , further comprising the step of providing a tracking mechanism on a device that captures the live images of the internal anatomical feature, and wherein said tracking information includes tracking information from said tracking mechanism.
24. The method of claim 22 , further comprising the step of providing an HMD live video feed capturing video of the patient observed by the user of the HMD, wherein said augmented reality view of the patient also includes images from the HMD live video feed.