Augmented reality guidance for spinal surgery and spinal procedures
Embodiments disclose a real-time surgery method and apparatus for displaying a stereoscopic augmented view of a patient from a static or dynamic viewpoint of the surgeon, which employs real-time three-dimensional surface reconstruction for preoperative and intraoperative image registration. Stereoscopic cameras provide real-time images of the scene including the patient. A stereoscopic video display is used by the surgeon, who sees a graphical representation of the preoperative or intraoperative images blended with the video images in a stereoscopic manner through a see through display.
1. A system comprising:
a stereoscopic optical see-through head mounted display;
a computing system configured to receive preoperative, intraoperative or preoperative and intraoperative image information of the physical spine of a patient and to generate a three-dimensional (3D) image of the spine of the patient,
one or more markers configured to be attached to a bony structure within a target portion of the physical spine of the patient,
wherein the computing system is configured to register the image information of the physical spine in a common coordinate system,
wherein the computing system is configured to register the one or more markers in the common coordinate system,
wherein the computing system is configured to generate a 3D stereoscopic display by the optical see-through head mounted display based on the 3D image of the spine,
wherein the computing system is configured to track movement from a first position to a second position of the physical spine during surgery,
wherein the computing system is configured to track the location and orientation of the optical see-through head mounted display in the common coordinate system,
wherein the computing system is configured to adjust in real time the 3D stereoscopic display based on the location and orientation of the optical see-through head mounted display,
wherein the computing system is configured to adjust in real time the 3D stereoscopic display based on the tracked movement from the first position to the second position of the physical spine,
wherein the computing system is configured to superimpose the 3D stereoscopic display on the target portion of the physical spine visible directly through the stereoscopic optical see-through head mounted display, and
wherein the first and second positions of the physical spine of the patient are different.
2. The system of claim 1 , wherein the computing system is configured to reconstruct one or more preoperative, intraoperative or preoperative and intraoperative two-dimensional (2D) images into three-dimensional images using 2D to 3D reconstruction.
3. The system of claim 1 , wherein the computing system is configured to track movement from the first position to the second position of the physical spine by tracking the position of the one or more markers in real time.
4. The system of claim 1 , wherein the computing system is configured to compare tracking data of instruments and/or devices to the tracked position of the physical spine during surgery in real time.
5. The system of claim 1 , wherein the computing system is configured to track the location and/or orientation of one or more of the instruments, devices, the physical spine, the surgeon, or the optical see-through head mounted display in real time.
6. The system of claim 5 , wherein the tracking of the one or more of the instruments, devices, the spine, the surgeon, or the optical see-through head mounted display is performed using one or more of attached passive optical markers, active optical markers, magnetic tracking, electromagnetic tracking, ultrasonic tracking, mechanical tracking, tracking using inertial measurement units and 3D scanners.
7. The system of claim 1 , wherein the computing system configured to receive preoperative, intraoperative or preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient is the same or different than one or more of the computing systems configured to register the image information of the spine in the common coordinate system, the computing system configured to register the one or more markers in the common coordinate system, the computing system configured to track movement from the first position to the second position of the physical spine during surgery, the computing system configured to track the location and orientation of the optical see-through head mounted display in the common coordinate system, the computing system configured to superimpose the 3D stereoscopic display displayed by the optical see-through head mounted display with the physical spine of the patient, and the computing system configured to adjust in real time the 3D stereoscopic display displayed by the optical see-through head mounted display.
8. The system of claim 1 , wherein the computing system configured to register the image information of the spine in the common coordinate system is the same or different than one or more of the computing systems configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computing system configured to register the one or more markers in the common coordinate system, the computing system configured to track movement from the first position to the second position of the physical spine during surgery, the computing system configured to track the location and orientation of the optical see-through head mounted display in the common coordinate system, the computing system configured to superimpose the 3D stereoscopic display displayed by the optical see-through head mounted display with the physical spine, and the computing system configured to adjust in real time the 3D stereoscopic display displayed by the optical see-through head mounted display.
9. The system of claim 1 , wherein the computing system configured to register the one or more markers in the common coordinate system is the same or different than one or more of the computing system configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computing systems configured to register the image information of the spine in the common coordinate system, the computing system configured to track movement from the first position to the second position of the physical spine during surgery, the computing system configured to track the location and orientation of the optical see-through head mounted display in the common coordinate system, the computing system configured to superimpose the 3D stereoscopic display displayed by the optical see-through head mounted display with the physical spine, and the computing system configured to adjust in real time the 3D stereoscopic display displayed by the optical see-through head mounted display.
10. The system of claim 1 , wherein the computing system configured to track movement from the first position to the second position of the physical spine during surgery is the same or different than one or more of the computing systems configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computing system configured to register the image information of the patient's spine in the common coordinate system, the computing system configured to register the one or more markers in the common coordinate system, the computing system configured to track the location and orientation of the optical see-through head mounted display in the common coordinate system, the computing system configured to superimpose the 3D stereoscopic display displayed by the optical see-through head mounted display with the physical spine, and the computing system configured to adjust in real time the 3D stereoscopic display displayed by the optical see-through head mounted display.
11. The system of claim 1 , wherein the computing system configured to track the location and orientation of the optical head see-through mounted display in the common coordinate system is the same or different than one or more of the computing systems configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computing system configured to register the image information of the spine in the common coordinate system, the computing system configured to register the one or more markers in the common coordinate system, the computing system configured to track movement from the first position to the second position of the physical spine during surgery, the computing system configured to superimpose the 3D stereoscopic display displayed by the optical see-through head mounted display with the physical spine, and the computing system configured to adjust in real time the 3D stereoscopic display displayed by the optical see-through head mounted display.
12. The system of claim 1 , wherein the computing system configured to adjust in real time the 3D stereoscopic display displayed by the optical see-through head mounted display is the same or different than one or more of the computing systems configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computing system configured to register the image information of the spine in the common coordinate system, the computing system configured to register the one or more markers in the common coordinate system, the computing system configured to track movement from the first position to the second position of the physical spine during surgery, and the computing system configured to track the location and orientation of the optical see-through head mounted display in the common coordinate system, and the computing system configured to superimpose the 3D stereoscopic display displayed by the optical see-through head mounted display with the physical spine.
13. The system of claim 1 , wherein the image information of the spine is registered with one or more structures of the physical spine using a navigation system.
14. The system of claim 1 , wherein the computing system is configured to provide for adjustment of transparency, color and contrast of the 3D stereoscopic display displayed by the optical see-through head mounted display superimposed on the physical spine.
15. The system of claim 1 , wherein the computing system is configured to display one or more virtual graphical representations of tracked instruments or devices by the optical see-through head mounted display superimposed on the physical spine.
16. The system of claim 1 , wherein the 3D stereoscopic display of the spine displayed by the optical see-through head mounted display includes 3D images of individually rendered vertebrae registered in the common coordinate system superimposed with individual vertebra of the physical spine.
17. The system of claim 16 , wherein the computing system is configured to track pose changes of the individual vertebrae in the common coordinate system and wherein the pose changes are translated into display of a different second 3D stereoscopic display at a location and orientation that corresponds to the location and orientation of the individual vertebrae of the spine after the pose changes.
18. The system of claim 1 , wherein the optical see-through head mounted display is configured to display one or more of graphical representations of instruments, surgical guides or techniques, or anatomical models superimposed on the physical spine.
19. The system of claim 1 , wherein the movement from the first position to the second position of the physical spine during surgery is a rotation of one or more vertebrae.
20. The system of claim 1 , wherein the target portion of the physical spine comprises an exposed portion of the physical spine.
21. The system of claim 1 , wherein the intraoperative image information is obtained from the patient using a CT scan.
22. A system comprising:
a stereoscopic optical see-through head mounted display;
a computing system configured to receive preoperative, intraoperative, or preoperative and intraoperative image information from a patient and to generate two or more 3D images of individualized vertebrae; and
one or more markers configured to be attached to a bony structure within a target portion of the physical spine of the patient,
wherein the computing system is configured to register the two or more 3D images of the individualized vertebrae in a common coordinate system,
wherein the computing system is configured to register the one or more markers in the common coordinate system,
wherein the computing system is configured to generate a 3D stereoscopic display by the optical see-through head mounted display based on the two or more 3D images of the individualized vertebrae,
wherein the computing system is configured to track movement from a first position to a second position of the physical spine during surgery by tracking the position of the one or more markers,
wherein the computing system is configured to track the location and orientation of the optical see-through head mounted display in the common coordinate system,
wherein the computing system is configured to adjust in real time the 3D stereoscopic display of the two or more 3D images of the individualized vertebrae based on the location and orientation of the optical see-through head mounted display,
wherein the computing system is configured to adjust in real time the 3D stereoscopic display of the two or more 3D images of the individualized vertebrae based on the tracked movement from the first position to the second position of the physical spine,
wherein the computing system is configured to superimpose the 3D stereoscopic display of the two or more 3D images of the individualized vertebrae on corresponding individual vertebrae of the physical spine visible directly through the stereoscopic optical see-through head mounted display, and
wherein the first and second positions of the physical spine of the patient are different.
23. The system of claim 22 , wherein the computing system is configured to detect changes in the sagittal position of the physical spine and to adjust the display of the two or more 3D images of the individualized vertebrae based on the detected changes in sagittal position.
24. The system of claim 22 , wherein the computing system is configured to detect changes in rotation of the physical spine and to adjust the display of the two or more 3D images of the individualized vertebrae based on the detected changes in rotation of the physical spine.
25. The system of claim 22 , wherein the location and orientation of the two or more 3D images of the individualized vertebrae displayed by the optical see-through head mounted display matches the location and orientation of the individual vertebrae of the physical spine.
26. The system of claim 22 , wherein the computing system is configured to track position changes of the individual vertebrae from the first position to the second position in the common coordinate system and wherein the position changes are translated into a different, second 3D stereoscopic display of 3D images of the individualized vertebrae corresponding to the second position displayed by the optical see-through head mounted display at a location and orientation that matches the location and orientation of the individual vertebrae of the physical spine after the position change.
27. The system of claim 22 , wherein the movement from a first position to a second position of the physical spine during surgery is a change in lateralization of the physical spine and wherein the computing system is configured to adjust the 3D stereoscopic display of the two or more 3D images of the individualized vertebrae based on the change in lateralization of the physical spine.
28. The system of claim 22 , wherein the image information from the patient is from a preoperative CT scan obtained with the patient in supine position, wherein the position of the physical spine during the surgery is a prone position, wherein the computing system is configured to generate two or more 3D images of individualized vertebrae based on the image information for 3D stereoscopic display by the optical see-through head mounted display, and wherein the computing system is configured to adjust the 3D stereoscopic display of the two or more 3D images of the individualized vertebrae based on the prone position of the physical spine during surgery to superimpose the 3D stereoscopic display of the two or more 3D images of the individualized vertebrae on corresponding individual vertebrae of the physical spine by the optical see-through head mounted display.
29. A system comprising:
a stereoscopic optical see-through head mounted display;
a computing system configured to receive preoperative, intraoperative, or preoperative and intraoperative image information from a patient and to generate two or more 3D images of two or more individualized vertebrae for stereoscopic display by the optical see-through head mounted display; and
two or more markers configured to be attached to two or more bony structures within a target portion of the physical spine of the patient,
wherein the computing system is configured to register the two or more 3D images of the two or more individualized vertebrae in a common coordinate system,
wherein the computing system is configured to register the two or more markers in the common coordinate system,
wherein the computing system is configured to generate a 3D stereoscopic display by the optical see-through head mounted display based on the two or more 3D images of the two or more individualized vertebrae,
wherein the computing system is configured to track movement from a first position to a second position of two or more individual vertebrae of the physical spine during surgery by tracking the position of the two or more markers,
wherein the computing system is configured to track the location and orientation of the optical see-through head mounted display in the common coordinate system,
wherein the computing system is configured to adjust in real time the 3D stereoscopic display based on the location and orientation of the optical see-through head mounted display,
wherein the computing system is configured to adjust in real time the 3D stereoscopic display of the two or more 3D images of the two or more individualized vertebrae based on the tracked movement from the first position to the second position of the corresponding individual vertebrae of the physical spine,
wherein the computing system is configured to superimpose the 3D stereoscopic display of the two or more 3D images of the two or more individualized vertebrae on the corresponding individual vertebrae of the physical spine visible directly through the stereoscopic optical see-through head mounted display, and
wherein the first and second positions of the physical spine of the patient are different.