Systems and methods for assisted surgical navigation
In at least one embodiment, a method of surgical navigation is provided. The method includes receiving an external three-dimensional model of a surgical site from the viewpoint of a headset, wherein the external three-dimensional model is derived from reflected light. The method further includes aligning the external three-dimensional model with an internal three-dimensional model of the surgical site from the viewpoint of the headset, wherein the internal three-dimensional model is derived from medical imaging, and generating an aligned view. The method further includes providing the aligned view to the headset, and updating the aligned view in real-time while the headset is moved or the surgical site is moved or modified during a surgical procedure.
1. A method, comprising: receiving, by a computing system, an external three-dimensional model of a surgical site from a headset, wherein the external three-dimensional model is produced by: a projector mounted to the headset projecting an array of non-visible light beams onto the surgical site, a camera mounted to the headset collecting light reflected from the surgical site back towards the headset to produce optical data regarding the surgical site, a laser pointer to light up reference landmarks on anatomical reference features, and a processor mounted to the headset, the processor configured to: construct a non-video-based external three-dimensional model based on the optical data, and produce mapping points by associating verbal annotations provided as the laser pointer lights up respective reference landmarks; fusing, by the computing system, the external three-dimensional model with an internal three-dimensional model of the surgical site from the viewpoint of the headset, wherein the internal three-dimensional model is derived from a medical imaging process; registering the internal three-dimensional model with the mapping points; generating, by the computing system, an aligned view of the surgical site; providing, by the computing system, the aligned view of the surgical site to the headset; and updating, by the computing system, the aligned view of the surgical site in real-time using the mapping points.
2. The method of claim 1 , wherein the array of non-visible light beams comprises infrared light.
3. The method of claim 1 , wherein the external three-dimensional model comprises a wireframe model of the surgical site.
4. The method of claim 1 , wherein the medical imaging process includes at least one of the following: computerized tomography (CT) scanning, magnetic resonance imaging (MRI), x-ray imaging, or ultrasound imaging.
5. The method of claim 1 , wherein the internal three-dimensional model is segmented according to relevant anatomy therein, thereby deriving a library of segment-relevant anatomical elements.
6. The method of claim 5 , wherein the aligned view shows segment-relevant anatomical elements in an anatomically correct correspondence with the surgical site.
7. The method of claim 1 , further comprising:
receiving, by the computing system, coordinates to indicate a position and orientation of the headset relative to the surgical site.
8. The method of claim 7 , wherein the coordinates indicating the position and orientation of the headset are derived from radio tracking that utilizes at least one fixed radio beacon.
9. The method of claim 7 , wherein the coordinates indicating the position and orientation of the headset are derived from optical tracking that utilizes at least one fixed optical beacon.
10. The method of claim 7 , wherein the coordinates indicating the position and orientation of the headset are derived from inertial guidance provided by an accelerometer associated with the headset.
11. The method of claim 7 , wherein updating the aligned view of the surgical site in real-time further comprises aligning the coordinates indicating the position and orientation of the headset with the internal three-dimensional model.
12. The method of claim 1 , further comprising:
receiving, by the computing system, a hand gesture relative to the surgical site; and
encode the hand gesture into an instruction to manipulate a portion of the aligned view of the surgical site.
13. The method of claim 12 , wherein the hand gesture is derived from radio tracking that utilizes at least one radio-reflective patch or radio-frequency identification (RFID) chip inside a glove.
14. The method of claim 12 , wherein the hand gesture is derived from optical tracking that utilizes structured light projected onto a glove.
15. The method of claim 12 , further comprising:
manipulating, by the computing system, an anatomical element in the internal three-dimensional model as a function of the encoded hand gesture; and
updating, by the computing system, the aligned view of the surgical site in real-time to show the manipulation of the anatomical element.
16. The method of claim 12 , wherein the hand gesture includes at least one of the following: a select command, an isolate command, a levitate command, a rotate command, a stop command, a zoom command, a measure command, a slice command, or a cross-section command.
17. The method of claim 1 , further comprising:
receiving, by the computing system, a surgical instrument position and orientation relative to the surgical site, or a sensor output of the surgical instrument.
18. The method of claim 17 , wherein the instrument position and orientation are derived from radio tracking that utilizes at least one radio-reflective patch or radio-frequency identification (RFID) chip on the instrument.
19. The method of claim 17 , further comprising:
updating, by the computing system, the aligned view of the surgical site in real-time to show the instrument position and orientation relative to the internal three-dimensional model or the sensor output of the surgical instrument.
20. The method of claim 19 , wherein the instrument position or orientation or sensor output includes at least one of the following: depth, angle, relative angle, relative elevation, volume, temperature, pressure, oxygenation or enervation.
21. The method of claim 1 , wherein a reflection of the array of non-visible light beams is captured by one or more cameras associated with the headset.
22. The method of claim 1 , further comprising:
recording, by the computing system, the aligned view of the surgical site for streaming or playback.
23. An article of manufacture including a non-transitory computer-readable medium having instructions stored thereon that, in response to execution by a computer system, cause the computer system to perform operations comprising: receiving an external three-dimensional model of a surgical site from a headset, wherein the external three-dimensional model is produced by: a projector mounted to the headset projecting an array of non-visible light beams onto the surgical site, a camera mounted to the headset collecting light reflected from the surgical site back towards the headset to produce optical data regarding the surgical site, a laser pointer to light up reference landmarks on anatomical reference features, and a processor mounted to the headset, the processor configured to: construct a non-video-based external three-dimensional model based on the optical data, and produce mapping points by associating verbal annotations provided as the laser pointer lights up respective reference landmarks; fusing the external three-dimensional model with an internal three-dimensional model of the surgical site from the viewpoint of the headset, wherein the internal three-dimensional model is derived from a medical imaging process; registering the internal three-dimensional model with the mapping points; generating an aligned view of the surgical site; providing the aligned view of the surgical site to the headset; and updating the aligned view of the surgical site in real-time.
24. A system, comprising: a processor; and a non-transitory memory having instructions stored thereon that, in response to execution by the processor, cause the system to perform operations comprising: receiving an external three-dimensional model of a surgical site from a headset, wherein the external three-dimensional model is produced by: a projector mounted to the headset projecting an array of non-visible light beams onto the surgical site, a camera mounted to the headset collecting light reflected from the surgical site back towards the headset to produce optical data regarding the surgical site, a laser pointer to light up reference landmarks on anatomical reference features, and a processor mounted to the headset, the processor configured to: construct a non-video-based external three-dimensional model based on the optical data, and produce mapping points by associating verbal annotations provided as the laser pointer lights up respective reference landmarks fusing the external three-dimensional model with an internal three-dimensional model of the surgical site from the viewpoint of the headset, wherein the internal three-dimensional model is derived from a medical imaging process; registering the internal three-dimensional model with the mapping points; generating an aligned view of the surgical site; providing the aligned view of the surgical site to the headset; and updating the aligned view of the surgical site in real-time.