Augmented reality headset for navigated robotic surgery

1. A surgical system comprising:

an augmented reality (AR) headset including a display screen configured to:

display images for viewing by a user and to allow ambient light to pass therethrough for viewing an anatomical structure by the user, and

a tracking system configured to determine a pose of an anatomical structure and a pose of a surgical tool;

at least one navigation controller configured to store a target pose for the surgical tool based on a surgical plan defining where a surgical procedure is to be performed using the surgical tool on the anatomical structure and based on the pose of the anatomical structure, and

at least one AR headset controller operatively connected to the at least one navigation controller and to the display screen, the at least one AR headset controller configured to simultaneously display a graphical representation of the anatomical structure at a position away from the physical anatomical structure, a graphical representation of the pose of the surgical tool on the displayed graphical representation of the anatomical structure and a graphical representation of the target pose for the surgical tool on the displayed graphical representation of the anatomical structure, the displayed graphical representation of the target pose being fixed relative to the graphical representation of the anatomical structure while the graphical representation of the pose of the surgical tool is updated on the display screen as the user manipulates the surgical tool;

wherein the display screen is configured with at least two laterally extending bands, wherein an upper one of the at least two laterally extending bands has a lower light transmissivity than a lower one of the at least two laterally extending bands to reduce the amount of incident ambient light passing therethrough for viewing by the user.

2. The surgical system of claim 1 , wherein the at least one AR controller is further configured to display two dimensional images in the upper one of the at least two laterally extending bands and to display a three dimensional model of the anatomical structure provided by the at least one navigation controller in the lower one of the at least two laterally extending bands.

3. The surgical system of claim 1 , wherein:

the AR headset further includes a gesture sensor configured to output to the at least one AR headset controller an indication of a hand gesture formed by the user; and

the at least one AR headset controller is configured to control graphical objects displayed on the display screen responsive to recognition of the hand gesture.

4. The surgical system of claim 3 , wherein:

the gesture sensor includes a camera mounted to face away from the user to output a video stream of the user's hand while positioned in front of the AR headset within a field-of-view of the camera; and

the at least one AR headset controller is configured to associate different recognizable hand gestures to different ones of a plurality of operations that control the graphical objects displayed on the display screen, wherein the plurality of operations includes at least one of the following: user command to expand a menu item that is displayed on the display screen into a list of sub-menu items that are then displayed on the display screen; user command to select a menu item and/or one of a plurality of sub-menu item that is displayed on the display screen; and user command to operationally control size, location, and/or orientation of one of the graphical objects that is displayed on the display screen.

5. The surgical system of claim 4 , wherein:

the at least one AR headset controller is configured to associate a recognizable hand gesture to a command to select a surgical implant from among a set of surgical implants that can be used for implantation into the anatomical structure, and to display a graphical representation of the surgical implant that is select on the display screen.

6. The surgical system of claim 3 , wherein:

the gesture sensor includes a camera mounted to face away from the user to output a video stream of the user's hand while positioned in front of the AR headset within a field-of-view of the camera; and

the at least one AR headset controller is configured to associate different recognizable hand gestures to different ones of a plurality of operations by the surgical system, wherein the plurality of operations includes at least one of the following: command to control imaging of the anatomical structure by an imaging system; and command to control which of a plurality of images of the anatomical structure is displayed on the display screen.

7. The surgical system of claim 3 , wherein:

the gesture sensor includes a camera mounted to face away from the user to output a video stream of the user's hand while positioned in front of the AR headset within a field-of-view of the camera; and

the at least one AR headset controller is configured to display a computer generated hand model on the display screen with a pose that is dynamically updated to track a present pose of the user's hand identified in the video stream from the camera.

8. The surgical system of claim 1 , wherein:

the AR headset further includes a microphone configured to output an audio stream to the at least one AR headset controller; and

the at least one AR headset controller is configured to control graphical objects displayed on the display screen responsive to recognition of voice commands contained in the audio stream.

9. The surgical system of claim 1 , wherein: the AR headset further includes a plurality of cameras spaced apart and mounted to face away from the user; and the tracking system is operative connected to process video streams from the plurality of cameras and configured to determine the pose of the anatomical structure and the pose of the surgical tool.

10. The surgical system of claim 9 , wherein the tracking system is configured to process the video streams from the plurality of cameras to determine pose of a reference array connected to the anatomical structure and pose of a reference array connected to the surgical tool.

11. The surgical system of claim 10 , further comprising a navigation camera that is not mounted to the AR headset,

wherein the tracking system is further configured to process a video stream from the navigation camera to determine pose of a reference array connected to the AR headset, and

wherein the at least one AR headset controller is configured to control a pose and scale of a three dimensional model of the anatomical structure that is displayed on the display screen based on the pose of the reference array connected to the anatomical structure and the pose of the reference array connected to the AR headset.

12. The surgical system of claim 11 , further comprising a pose sensor mounted to the AR headset and configured to output a sensed pose of the AR headset,

wherein the at least one AR headset controller is operatively connected to the pose sensor and further configured to control a pose of a three dimensional model of the anatomical structure that is displayed on the display screen based on the pose of the reference array connected to the anatomical structure and the sensed pose of the AR headset.

13. The surgical system of claim 1 , further comprising a first AR headset and a second AR headset,

wherein each of the first and second AR headsets include a camera,

wherein the least one AR headset controller is operatively connected to receive a first video stream from the camera of the first AR headset and configured to selective route the first video stream to the display device of the first AR headset and/or to the display device of the second AR headset responsive to electronic commands obtained from a user wearing the first AR headset, and

wherein the least one AR headset controller is operatively connected to receive a second video stream from the camera of the second AR headset and configured to selective route the second video stream to the display device of the first AR headset and/or to the display device of the second AR headset responsive to electronic commands obtained from a user wearing the second AR headset.

14. The surgical system of claim 1 , wherein the at least one AR headset controller is further configured to determine a trajectory offset distance between where a planned trajectory of the target pose for the surgical tool intersects a target location on the anatomical structure and where a trajectory of the pose of the surgical tool intersects the target location on the anatomical structure.

15. The surgical system of claim 14 , wherein the at least one AR headset controller is configured to display each of the graphical representations of the target pose for the surgical tool and the pose of the surgical tool as elongated objects each having a major axis extending between opposite ends, and control spacing between where one end of the elongated objects are displayed on the display screen responsive to the trajectory offset distance.

16. The surgical system of claim 15 , wherein the at least one AR headset controller is configured to control an angle of incline between the elongated objects displayed on the display screen responsive to an angle between the planned trajectory of the target pose for the surgical tool and the trajectory of the pose of the surgical tool.

17. The surgical system of claim 16 , wherein the at least one AR headset controller is configured to display each of the graphical representations of the target pose for the surgical tool and the pose of the surgical tool as circular objects and control spacing between centers of the circular objects displayed on the display screen responsive to the trajectory offset distance.

18. A surgical system comprising:

an augmented reality (AR) headset including a display screen configured to display images for viewing by a user and to allow ambient light to pass therethrough for viewing a physical anatomical structure by the user, and

a tracking system configured to determine a pose of the anatomical structure and a pose of a surgical tool;

at least one navigation controller configured to store a target pose for the surgical tool based on a surgical plan defining where a surgical procedure is to be performed using the surgical tool on the anatomical structure and based on the pose of the anatomical structure, and

at least one AR headset controller operatively connected to the at least one navigation controller and to the display screen, the at least one AR headset controller configured to simultaneously display a graphical representation of the anatomical structure at a position away from the physical anatomical structure, a graphical representation of the pose of the surgical tool on the displayed graphical representation of the anatomical structure and a graphical representation of the target pose for the surgical tool on the displayed graphical representation of the anatomical structure, the displayed graphical representation of the target pose being fixed relative to the graphical representation of the anatomical structure while the graphical representation of the pose of the surgical tool is updated on the display screen as the user manipulates the surgical tool.