Augmented reality guidance for spinal procedures using stereoscopic optical see-through head mounted displays with display of virtual surgical guides
Embodiments disclose a real-time surgery method and apparatus for displaying a stereoscopic augmented view of a patient from a static or dynamic viewpoint of the surgeon, which employs real-time three-dimensional surface reconstruction for preoperative and intraoperative image registration. Stereoscopic cameras provide real-time images of the scene including the patient. A stereoscopic video display is used by the surgeon, who sees a graphical representation of the preoperative or intraoperative images blended with the video images in a stereoscopic manner through a see-through display.
1. A system comprising:
a stereoscopic optical see-through head mounted display;
a physical instrument or physical device;
at least one computing system;
at least one camera; and
at least one marker configured to be attached to a bony structure of a physical spine of a patient;
wherein the at least one computing system is configured to track, using the at least one camera, the at least one marker,
wherein the at least one computing system is configured to generate a three-dimensional (3D) surface representation of a virtual guide for the physical instrument or the physical device,
wherein the at least one computing system is configured to generate a 3D stereoscopic view, displayed by the optical see-through head mounted display, the 3D stereoscopic view comprising the 3D surface representation of the virtual guide for the physical instrument or the physical device,
wherein the system is configured to adjust in real time the 3D stereoscopic view responsive to movement of the optical see-through head mounted display,
wherein the system is configured to adjust in real time the 3D stereoscopic view responsive to movement of the physical spine from a first position to a second position,
wherein the at least one computing system is configured to superimpose the 3D stereoscopic view on a target portion of the physical spine, and
wherein the first and second positions of the physical spine of the patient are different.
2. The system of claim 1 , wherein the 3D surface representation of the virtual guide indicates a proposed measurement, bone removal, position, orientation, angle, or combinations thereof for the physical instrument or the physical device.
3. The system of claim 1 , wherein the virtual guide comprises a virtual trajectory for the physical instrument, a virtual trajectory for the physical device, a virtual pin, a virtual screw, a virtual nail, a virtual plate, a virtual template of the physical instrument, a virtual template of the physical device, or any combinations of the foregoing.
4. The system of claim 1 , wherein the 3D surface representation of the virtual guide is a graphical representation displayed over the target portion of the physical spine.
5. The system of claim 1 , wherein the target portion of the physical spine is a process, pedicle, lamina or posterior body of a vertebra.
6. The system of claim 5 , wherein the virtual guide is configured for positioning at least one transpedicular screw in relationship to the process, pedicle, lamina or posterior body of the vertebra.
7. The system of claim 1 , wherein the at least one camera is head mounted with the stereoscopic optical see-through head mounted display.
8. The system of claim 1 , wherein the at least one camera is separate from the stereoscopic optical see-through head mounted display.
9. The system of claim 1 , wherein the system comprises at least one inertial measurement unit, and wherein the system is configured to adjust in real time the at least one 3D stereoscopic view responsive to movement of the optical see-through head mounted display using the at least one inertial measurement unit.
10. The system of claim 1 , wherein the at least one computing system is configured to receive an image information of the physical spine of the patient, wherein the image information is a preoperative, intraoperative, or preoperative and intraoperative image information of the physical spine of the patient.
11. The system of claim 10 , wherein the at least one marker comprises at least one optical marker and at least one radiopaque marker, and wherein the at least one radiopaque marker is included in the image information of the physical spine of the patient.
12. The system of claim 11 , wherein the at least one computing system is configured to register the image information with the physical spine using the at least one radiopaque marker.
13. The system of claim 11 , wherein the at least one optical marker is a color marker, a reflective marker, or a color marker and a reflective marker.
14. The system of claim 1 , wherein the 3D stereoscopic view comprises a 3D surface representation of an anatomical model, an image information of the physical spine of the patient, anatomic data derived from the image information of the physical spine of the patient or a combination thereof.
15. The system of claim 14 , wherein the at least one computing system is configured to generate the 3D surface representation of the anatomical model, the image information of the physical spine of the patient, the anatomic data derived from the image information of the physical spine of the patient or the combination thereof based on at least one 2D image, at least one 3D image, or at least one 2D and at least one 3D image of the physical spine, and wherein the at least one 2D image, at least one 3D image or combination thereof comprises a CT image, an MRI image, an ultrasound image, an x-ray image or combinations thereof.
16. The system of claim 1 , wherein the at least one computing system is configured to determine in real time the relative location, orientation or location and orientation of the physical instrument, the physical device, the at least one marker, the physical spine, a surgeon, the optical see-through head mounted display, or any combinations of the foregoing.
17. The system of claim 1 , wherein one or more of the at least one computing system configured to track, using the at least one camera, the at least one marker, the at least one computing system configured to generate the-3D surface representation of the virtual guide for the physical instrument or the physical device, the at least one computing system configured to generate the 3D stereoscopic view, displayed by the optical see-through head mounted display, and the at least one computing system configured to superimpose the 3D stereoscopic view on the target portion of the physical spine are the same.
18. The system of claim 1 , wherein one or more of the at least one computing system configured to track, using the at least one camera, the at least one marker, the at least one computing system configured to generate the 3D surface representation of the virtual guide for the physical instrument or the physical device, the at least one computing system configured to generate the 3D stereoscopic view, displayed by the optical see-through head mounted display, the at least one computing system configured to superimpose the 3D stereoscopic view on the target portion of the physical spine are different.
19. The system of claim 14 , wherein the 3D surface representation of the anatomical model, the image information of the physical spine of the patient, the anatomic data derived from the image information of the physical spine of the patient or the combination thereof comprises a 3D surface representation of one or more individualized vertebrae.
20. The system of claim 1 , wherein at least one marker is attached to the physical instrument or physical device, and the computing system is configured to track the physical instrument or physical device.
21. The system of claim 1 , comprising at least one passive optical marker, active optical marker, inertial measurement unit, magnetic tracking device, electromagnetic tracking device, ultrasonic tracking device, mechanical tracking device, tracking device using inertial measurement units, 3D scanner, camera, navigation system or a combination thereof configured to track the physical instrument or the physical device, the physical spine, a surgeon, the optical see-through head mounted display or a combination thereof.
22. The system of claim 1 , wherein the movement from the first position to the second position of the physical spine during surgery is a rotation or a lateralization of at least one vertebrae.
23. The system of claim 1 , wherein the computing system is configured to obtain coordinate information of the physical instrument or the physical device, the at least one marker, the physical spine, a surgeon, the optical see-through head mounted display or a combination thereof in a coordinate system.
24. The system of claim 23 , wherein the coordinate system is the coordinate system of the at least one camera.
25. The system of claim 23 , wherein the coordinate system is the coordinate system of the at least one marker.
26. The system of claim 1 , wherein the at least one camera is configured to use visible light information.
27. The system of claim 1 , wherein the at least one camera is configured to use infrared light information.
28. The system of claim 1 , wherein the system is configured to generate the 3D surface representation of the virtual guide using at least a point cloud, mesh, or a combination thereof.
29. The system of claim 1 , wherein the computing system is configured to adjust the 3D stereoscopic view for the interpupillary distance of the surgeon.
30. The system of claim 1 , further comprising a 3D scanner, a video system or a 3D scanner and a video system configured to track the movement from the first position to the second position of the physical spine.