Surgical systems and methods for providing surgical guidance with a head-mounted device
A head-mounted device (HMD) for use with a navigation system. The navigation system includes a localizer and a bone tracker coupled to a physical bone. The localizer monitors the bone tracker to track a pose of the physical bone in a localizer coordinate system. Controller(s) obtain surgical plan data related to the physical bone. The surgical plan data includes a virtual model of the physical bone and cut volume data defining material to be removed from the physical bone in preparation for receiving an implant. The controller(s) register the HMD to the localizer coordinate system and provide, on the HMD display, computer-generated imagery representative of the surgical plan data, including the virtual model of the physical bone and the cut volume data. The controller(s) align and combine the computer-generated imagery with a real-world view of the physical bone.
1. A surgical system comprising:
a head-mounted device (HMD) comprising an HMD controller and a display that is configured to be located in front of eyes of a wearer of the HMD;
a navigation system comprising a localizer and a bone tracker coupled to a physical bone of a patient, wherein the localizer comprises a localizer coordinate system and is configured to monitor the bone tracker to track a pose of the physical bone in the localizer coordinate system, wherein the navigation system is configured to:
obtain surgical plan data related to the physical bone, wherein the surgical plan data comprises a virtual model of the physical bone, cut volume data defining material to be removed from the physical bone in preparation for receiving an implant, and a textual note previously created by a surgeon during surgical planning related to the physical bone, wherein the textual note is associated with a particular surgical step related to the physical bone;
register the HMD to the localizer coordinate system using the localizer such that the localizer is able to obtain a pose of the HMD; and
transmit the surgical plan data to the HMD controller; and
wherein the HMD controller is configured to:
provide, on the display, computer-generated imagery representative of the surgical plan data, including the virtual model of the physical bone and the cut volume data;
align and combine the computer-generated imagery with a real-world view of the physical bone; and
automatically present, on the display, the textual note at the time of the particular surgical step with which the textual note is associated.
2. The surgical system of claim 1 , wherein the HMD controller aligns the computer-generated imagery with the real-world view of the physical bone in accordance with the tracked pose of the physical bone and the pose of the HMD.
3. The surgical system of claim 1 , wherein the navigation system is configured to register the surgical plan data to the localizer coordinate system through combination of:
a first transform defining a relationship between the physical bone and the bone tracker; and
a second transform defining a relationship between the bone tracker and the localizer.
4. The surgical system of claim 1 , wherein:
the HMD further comprises a video camera;
the navigation system further comprises a registration device including a plurality of registration markers that are detectable by the localizer and by the video camera of the HMD; and
the navigation system is configured to register the HMD to the localizer coordinate system through combination of a first transform defining a relationship between the HMD and the registration device and a second transform a relationship between the registration device and the localizer.
5. The surgical system of claim 1 , wherein:
the navigation system comprises an HMD tracker coupled to the HMD; and
the navigation system is configured to register the HMD to the localizer coordinate system through detection of a pose of the HMD tracker.
6. The surgical system of claim 1 , wherein the HMD further comprises a video camera, and wherein the real-world view is further defined as a real-world video captured by the video camera of the HMD.
7. The surgical system of claim 1 , wherein:
the surgical plan data further comprises a size of the implant and/or a placement of the implant; and
the HMD controller is configured to further provide, on the display, computer-generated imagery representative of the surgical plan data, including the size of the implant and/or the placement of the implant.
8. The surgical system of claim 1 , further comprising a surgical tool that is configured to remove the material from the physical bone in preparation for receiving the implant, and wherein:
the navigation system further includes a tool tracker coupled to the surgical tool;
the navigation system is configured to monitor the tool tracker to track poses of the surgical tool in the localizer coordinate system; and
the HMD controller is configured to provide, on the display, computer-generated imagery representative of the surgical tool relative to the computer-generated imagery including the virtual model of the physical bone and the cut volume data.
9. The surgical system of claim 8 , wherein the cut volume data further comprises a virtual cutting boundary that defines the material to be removed from the physical bone, wherein the virtual cutting boundary is designed to prevent the surgical tool from removing material from an undesired area of the patient.
10. The surgical system of claim 9 , wherein the surgical tool is hand-held.
11. The surgical system of claim 1 , further comprising a manipulator comprising a base and a robotic arm extending from the base, wherein the robotic arm is configured to support and move a surgical tool that is configured to remove the material from the physical bone in preparation for receiving the implant, and wherein:
the navigation system further includes a tracker coupled to the manipulator;
the navigation system is configured to monitor the tracker to track a pose of the surgical tool in the localizer coordinate system; and
the HMD controller is configured to further provide, on the display, computer-generated imagery representative of the surgical tool relative to the computer-generated imagery including the virtual model of the physical bone and the cut volume data.
12. The surgical system of claim 11 , wherein the cut volume data further comprises a virtual cutting boundary that defines the material to be removed from the physical bone, wherein the virtual cutting boundary is designed to prevent the surgical tool from removing material from an undesired area of the patient.
13. The surgical system of claim 11 , wherein:
the surgical plan data further comprises a tool path along which the manipulator moves, or will move, the surgical tool to remove the material from the physical bone in preparation for receiving the implant; and
the HMD controller is configured to further provide, on the display, computer-generated imagery representative of the tool path relative to the computer-generated imagery including the virtual model of the physical bone and the cut volume data.
14. The surgical system of claim 1 , wherein the HMD further comprises a sensor configured to recognize gesture commands from the wearer of the HMD, and wherein the HMD controller is configured to detect the gesture commands to enable modification to the surgical plan data.
15. The surgical system of claim 1 , wherein the display of the HMD is further defined as an organic light-emitting diode display.
16. A method of operating a surgical system, the surgical system comprising a head-mounted device (HMD) including an HMD controller and a display that is configured to be located in front of eyes of a wearer of the HMD, a navigation system including a localizer and a bone tracker coupled to a physical bone of a patient, wherein the localizer comprises a localizer coordinate system and is configured to monitor the bone tracker to track a pose of the physical bone in the localizer coordinate system, the method comprising:
with the navigation system:
obtaining surgical plan data related to the physical bone, wherein the surgical plan data comprises a virtual model of the physical bone, cut volume data defining material to be removed from the physical bone in preparation for receiving an implant, and a textual note previously created by a surgeon during surgical planning related to the physical bone, wherein the textual note is associated with a particular surgical step related to the physical bone;
registering the HMD to the localizer coordinate system using the localizer such that the localizer is able to obtain a pose of the HMD; and
transmitting the surgical plan data to the HMD controller; and
with the HMD controller:
providing, on the display, computer-generated imagery representative of the surgical plan data, including the virtual model of the physical bone and the cut volume data;
aligning and combining the computer-generated imagery with a real-world view of the physical bone; and
automatically presenting, on the display, the textual note at the time of the particular surgical step with which the textual note is associated.
17. The method of claim 16 , comprising the HMD controller aligning the computer-generated imagery with the real-world view of the physical bone in accordance with the tracked pose of the physical bone and the pose of the HMD.
18. The method of claim 16 , comprising the navigation system registering the surgical plan data to the localizer coordinate system by combining:
a first transform defining a relationship between the physical bone and the bone tracker; and
a second transform defining a relationship between the bone tracker and the localizer.
19. The method of claim 16 , wherein the HMD further comprises a video camera and the navigation system further comprises a registration device including a plurality of registration markers that are detectable by the localizer and by the video camera of the HMD, the method further comprising the navigation system:
registering the HMD to the localizer coordinate system by combining a first transform defining a relationship between the HMD and the registration device and a second transform a relationship between the registration device and the localizer.
20. The method of claim 16 , wherein the navigation system comprises an HMD tracker coupled to the HMD, the method further comprising the navigation system registering the HMD to the localizer coordinate system through detection of a pose of the HMD tracker.
21. The method of claim 16 , wherein the HMD further comprises a video camera, the method comprising the HMD controller aligning and combining the computer-generated imagery with real-world video of the physical bone captured by the video camera of the HMD.
22. The method of claim 16 , wherein the surgical plan data further comprises a size of the implant and/or a placement of the implant, the method comprising the HMD controller further providing, on the display, computer-generated imagery representative of the surgical plan data, including the size of the implant and/or the placement of the implant.
23. The method of claim 16 , further comprising a surgical tool that is configured to remove the material from the physical bone in preparation for receiving the implant, and wherein the navigation system further includes a tool tracker coupled to the surgical tool, the method further comprising:
the navigation system monitoring the tool tracker to track poses of the surgical tool in the localizer coordinate system; and
the HMD controller further providing, on the display, computer-generated imagery representative of the surgical tool relative to the computer-generated imagery including the virtual model of the physical bone and the cut volume data.
24. The method of claim 16 , further comprising a manipulator comprising a base and a robotic arm extending from the base, wherein the robotic arm is configured to support and move a surgical tool that is configured to remove the material from the physical bone in preparation for receiving the implant, and wherein the navigation system further includes a tracker coupled to the manipulator, the method further comprising:
the navigation system monitoring the tracker to track a pose of the surgical tool in the localizer coordinate system; and
the HMD controller further providing, on the display, computer-generated imagery representative of the surgical tool relative to the computer-generated imagery including the virtual model of the physical bone and the cut volume data.
25. The method of claim 24 , wherein the surgical plan data further comprises a tool path along which the manipulator moves, or will move, the surgical tool to remove the material from the physical bone in preparation for receiving the implant, the method further comprising:
the HMD controller further providing, on the display, computer-generated imagery representative of the tool path relative to the computer-generated imagery including the virtual model of the physical bone and the cut volume data.
26. The method of claim 16 , wherein the HMD further comprises a sensor configured to recognize gesture commands from the wearer of the HMD, the method further comprising the HMD controller detecting the gesture commands to enable modifying the surgical plan data.
27. A head-mounted device (HMD) for use with a navigation system, the navigation system including a localizer and a bone tracker coupled to a physical bone of a patient, wherein the localizer comprises a localizer coordinate system and is configured to monitor the bone tracker to track a pose of the physical bone in the localizer coordinate system, the HMD comprising:
an HMD controller; and
a display that is configured to be located in front of eyes of a wearer of the HMD; and
wherein the HMD controller is configured to:
receive, from the navigation system, surgical plan data related to the physical bone, wherein the surgical plan data comprises a virtual model of the physical bone, cut volume data defining material to be removed from the physical bone in preparation for receiving an implant, and a textual note previously created by a surgeon during surgical planning related to the physical bone, wherein the textual note is associated with a particular surgical step related to the physical bone;
register to the localizer coordinate system;
provide, on the display, computer-generated imagery representative of the surgical plan data, including the virtual model of the physical bone and the cut volume data;
align and combine the computer-generated imagery with a real-world view of the physical bone; and
automatically present, on the display, the textual note at the time of the particular surgical step with which the textual note is associated.
28. The HMD of claim 27 , further comprising a video camera, and wherein the HMD controller is configured to register to the localizer coordinate system through detection, with the video camera, of registration markers of a registration device, wherein the registration markers are also detectable by the localizer.
29. The HMD of claim 27 , further comprising an HMD tracker coupled to the HMD, wherein the HMD tracker is configured to be detected by the localizer and to enable the HMD controller to register to the localizer coordinate system.
30. The HMD of claim 27 , wherein the HMD further comprises a video camera, and wherein the real-world view is further defined as a real-world video captured by the video camera of the HMD.
31. The HMD of claim 27 , wherein:
the surgical plan data further comprises a size of the implant and/or a placement of the implant; and
the HMD controller is configured to further provide, on the display, computer-generated imagery representative of the surgical plan data, including the size of the implant and/or the placement of the implant.
32. The HMD of claim 27 , wherein the HMD controller is configured to provide, on the display, computer-generated imagery representative of a surgical tool relative to the computer-generated imagery including the virtual model of the physical bone and the cut volume data.
33. The HMD of claim 27 , wherein the cut volume data further comprises a virtual cutting boundary that defines the material to be removed from the physical bone.
34. The HMD of claim 27 , wherein the HMD controller is configured to further provide, on the display, computer-generated imagery representative of a tool path relative to the computer-generated imagery including the virtual model of the physical bone and the cut volume data.
35. The HMD of claim 27 , further comprising a sensor configured to recognize gesture commands from the wearer, and wherein the HMD controller is configured to detect the gesture commands to enable modification to the surgical plan data.
36. The HMD of claim 27 , wherein the display is further defined as an organic light-emitting diode display.
37. A head-mounted device (HMD) for providing surgical guidance related to a physical bone of a patient, the HMD comprising:
an HMD controller;
a video camera; and
a display that is configured to be located in front of eyes of a wearer of the HMD; and
wherein the HMD controller is configured to:
receive surgical plan data related to the physical bone, wherein the surgical plan data comprises a virtual model of the physical bone, cut volume data defining material to be removed from the physical bone in preparation for receiving an implant, and a textual note previously created by a surgeon during surgical planning related to the physical bone, wherein the textual note is associated with a particular surgical step related to the physical bone;
register to the physical bone;
provide, on the display, computer-generated imagery representative of the surgical plan data, including the virtual model of the physical bone and the cut volume data;
align and combine the computer-generated imagery with a real-world video of the physical bone captured by the video camera; and
automatically present, on the display, the textual note at the time of the particular surgical step with which the textual note is associated.