Immersive three-dimensional display for robotic surgery

1. An immersive display for use in a robotic surgical system, comprising:

a support arm;

a housing mounted to the support arm and configured to provide a three-dimensional display of at least one image from an endoscopic camera used in the robotic surgical system;

at least one sensor configured to detect a head gesture of the user; and

a controller that configures the three-dimensional display to display a modified image from the endoscopic camera in response to the at least one sensor detecting an intentional head gesture of the user, wherein the controller configures the immersive display to maintain a relative spatial relationship of the housing and a hand of the user by providing visual cues to position the hand of the user at a target location in space.

2. The immersive display of claim 1 , wherein the controller modifies an image from the endoscopic camera based on input from a sensor or a control operating as a clutch.

3. The immersive display of claim 2 wherein the input, from the sensor or the control that is operating as a clutch, is generated responsive to depressing a foot pedal or holding onto a handle with sensors.

4. The immersive display of claim 1 , wherein the housing is coupled to a contoured face frame configured to engage with a face of the user.

5. The immersive display of claim 1 , wherein the relative spatial relationship is a first relative spatial relationship, wherein the immersive display provides visual cues for maintaining a correspondence between i) the first relative spatial relationship of the housing and the at least one user hand position and ii) a second relative spatial relationship of the endoscopic camera and a surgical instrument.

6. The immersive display of claim 1 , wherein in response to the at least one sensor detecting a forward head gesture, the controller configures the three-dimensional display to display a zoomed-in image from the endoscopic camera, and wherein in response to the sensor detecting a backward head gesture, the three-dimensional display is configured to display a zoomed-out image from the endoscopic camera.

7. The immersive display of claim 1 , wherein in response to the sensor detecting a lateral head gesture, the controller configures three-dimensional display to display a panning image from the endoscopic camera, and wherein in response to the sensor detecting a tilting head gesture, the three-dimensional display is configured to display a tilting image from the endoscopic camera.

8. The immersive display of claim 1 , wherein the controller configures the three-dimensional display to display a graphical user interface and uses the at least one sensor to detect a head gesture to navigate the graphical user interface.

9. The immersive display of claim 1 , wherein the controller configures the three-dimensional display to display a visual representation of at least one of a user hand position and a user foot position relative to at least one target position.

10. The immersive display of claim 1 , further comprising at least one external camera coupled to the housing, wherein the camera is configured to provide at least one image of an environment external to the housing.

11. A method for operating a robotic surgical system that includes a three-dimensional display, the method comprising:

providing at least one image from an endoscopic camera used in the robotic surgical system, for three-dimensional display to a user;

detecting using at least one sensor a head gesture of the user;

configuring the three-dimensional display to display a modified image from the endoscopic camera in response to detecting an intentional head gesture of the user; and

maintaining a relative spatial relationship of the three-dimensional display and a hand of the user by providing visual cues to position the hand of the user at a target location in space.

12. The method of claim 11 wherein configuring the three-dimensional display is based on the user activating a clutch and leaning in, to command a camera view zoom in.

13. The method of claim 12 further comprising configuring the three-dimensional display to display a further modified image from the endoscopic camera in response to detecting a further intentional head gesture of the user when the user leans out, to command a camera view zoom out.

14. The method of claim 11 wherein configuring the three-dimensional display is based on the user activating a clutch and the user leaning out, to command a camera view zoom out.

15. The method of claim 11 wherein configuring the three-dimensional display is based on the user activating a clutch and turning their head left or right to command a camera view pan left or a camera view pan right.

16. The method of claim 11 wherein configuring the three-dimensional display is based on the user activating a clutch and tilting their head forward or backward to command a camera view tilt forward or a camera view tilt backward.

17. The method of claim 11 wherein the user's hand is holding a handheld user interface devices, while simultaneously controlling the display of modified images from the endoscopic camera via their head gestures.

18. The method of claim 11 further comprising configuring the three-dimensional display to display a graphical user interface and using the at least one sensor to detect a head gesture to navigate the graphical user interface.

19. The immersive display of claim 1 further comprising an outward-facing illuminator, wherein the immersive display provides visual cues by configuring the outward-facing illuminator to project one or more reference indicators to highlight the target location in space.

20. The immersive display of claim 1 , wherein the immersive display provides visual cues by displaying a graphical representation on the three-dimensional display for where the user's hand is to be positioned with respect to a current housing location and orientation.

21. The immersive display of claim 5 , wherein the robotic surgical system comprises a handheld user interface device (UID) that is configured to control the surgical instrument, wherein the first and second relative spatial relationships enable the robotic surgical system to perform a teleoperation during which a relative position of the user's head and the user's hands that are each holding a handheld UID is substantially similar to a relative position of the endoscopic camera and the surgical instrument.

22. The method of claim 11 , wherein the relative spatial relationship is a first relative spatial relationship, wherein method further comprises configuring the three-dimensional display to provide guidance using the provided visual cues for maintaining a correspondence between i) the first relative spatial relationship of the three-dimensional display and user hand positions and ii) a second relative spatial relationship of the endoscopic camera and a surgical instrument.

23. The method of claim 11 , wherein the robotic surgical system further comprises an outward-facing illuminator, wherein the visual cues are provided by configuring the outward-facing illuminator to project one or more reference indicators to highlight user hand positions in space.

24. The method of claim 11 , wherein the visual cues are provided by displaying a graphical representation for where a user's hand is to be positioned with respect to a current location and orientation of the three-dimensional display.

25. The method of claim 22 , wherein the robotic surgical system comprises a handheld user interface device (UID) that is configured to control the surgical instrument, wherein the first and second relative spatial relationships enable the robotic surgical system to perform a teleoperation during which a relative position of the user's head and the user's hands that are each holding a handheld UID is substantially similar to a relative position of the endoscopic camera and the surgical instrument.