Surgeon head-mounted display apparatuses
An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.
1. An augmented reality (AR) surgical system comprising:
a display having a partially transparent display screen that displays images while allowing transmission of ambient light therethrough to allow a user to look through the display screen to view objects:
a computer equipment coupled to the display and configured to:
generate a three dimensional (3D) anatomical model using patient data created by medical imaging equipment that has imaged a portion of the patient;
rotate and scale the 3D anatomical model;
generate a video signal based on the rotated and scaled anatomical model; and
output a video signal to the display screen;
the computer equipment including a gesture interpretation module configured to sense gestures of a user and to change the displayed video signal on the display based on the sensed gestures of the user.
2. The AR surgical system of claim 1 , wherein the gesture interpretation module is configured to recognize a gesture as corresponding to one of a plurality of defined commands and trigger operation of the corresponding command.
3. The AR surgical system of claim 1 , wherein the gesture interpretation module is configured to sense movement of at least one finger of the user.
4. The AR surgical system of claim 3 , wherein the computer equipment is configured to adjust the displayed video signal on the display screen based on the sensed finger movement of the user.
5. The AR surgical system of claim 3 , further comprising a camera mounted to the display, wherein the gesture interpretation module is configured to sense movement of a hand of the user through the camera.
6. The AR surgical system of claim 1 , wherein the computer equipment is configured to position the displayed video signal to appear to hover in space above and adjacent to the location of the operation site.
7. The AR surgical system of claim 1 , wherein the display screen includes a coating that provides variable contrast.
8. The AR surgical system of claim 1 , wherein the display screen includes a variable contrast coating to enhance viewability of information displayed on the display screen.
9. An augmented reality (AR) surgical system comprising:
a display including a see-through display screen that display images while allowing transmission of ambient light therethrough; and
a camera connected to the display and configured to observe reference markers connected to a patient, and reference markers connected to a surgical tool located within a surgical room;
computer equipment configured to:
compute the relative location and orientation of the headset and the reference markers connected to the patient based on processing an image signal from the camera;
generate a three dimensional (3D) anatomical image using patient data created by medical imaging equipment that has imaged a portion of the patient;
rotate and scale the 3D anatomical model based on the relative location and orientation of the reference markers connected to the display and the reference markers connected to the patient; and
generate a video signal based on the 3D anatomical image, and the location and orientation of the reference makers coupled to the patient and the surgical tool,
output the video signal to the display screen;
the computer equipment including a gesture interpretation module configured to sense gestures of a user and to change the displayed video signal on the display based on the sensed gestures of the user.
10. The AR surgical system of claim 9 , wherein the gesture interpretation module is configured to recognize a gesture as corresponding to one of a plurality of defined commands and trigger operation of the corresponding command.
11. The AR surgical system of claim 9 , wherein the gesture interpretation module is configured to sense movement of at least one finger of the user.
12. The AR surgical system of claim 11 , wherein the computer equipment is configured to adjust the displayed video signal on the display screen based on the sensed finger movement of the user.
13. The AR surgical system of claim 11 , further comprising a camera mounted to the display, wherein the gesture interpretation module is configured to sense movement of a hand of the user through the camera.
14. The AR surgical system of claim 9 , wherein the computer equipment is configured to position the displayed video signal to appear to hover in space above and adjacent to the location of the operation site.
15. The AR surgical system of claim 9 , wherein the display screen includes a coating that provides variable contrast.
16. The AR surgical system of claim 9 , wherein the display screen includes a variable contrast coating to enhance viewability of information displayed on the display screen.
17. The AR surgical system of claim 9 , wherein the computer equipment is further configured to generate a graphical representation of a virtual trajectory extending from the surgical tool into the 3D anatomical model based on the relative location and orientation of the reference markers connected to the display, the reference markers connected to the patient, and the reference markers connected to the surgical tool, and generate the video signal to include the graphical representation of the virtual trajectory.
18. The AR surgical system of claim 17 , wherein the computer equipment is further configured to select an image slice among a plurality of image slices contained in the 3D anatomical model, the image slice being selected based on the computer equipment determining that the image slice is traversed by the virtual trajectory extending from the surgical tool, and to generate the video signal to include the image slice.
19. The AR surgical system of claim 18 , wherein a graphical representation of the image slice and the graphical representation of the virtual trajectory that are displayed on the display screen are responsive to a head motion signal from a motion sensor coupled to the display.
20. The AR surgical system of claim 18 , wherein a graphical representation of the image slice and the graphical representation of the virtual trajectory that are displayed on the display screen provide a view of the image slice that is perpendicular to a direction of the virtual trajectory.