Methods and systems for touchless control of surgical environment

1. A system for touchless control of one or more medical equipment devices in an operating room, the system comprising:

a 3D optical sensor connected for detecting one or more gestures made by a practitioner in a sensing volume of the sensor;

a controller connected to receive 3D optical data from the 3D optical sensor and configured to provide a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more hand gestures made by the practitioner in a volumetric spatial region corresponding to the menu item in the sensing volume; and

one or more interactive display unit (IDU) displays for displaying, outside of the sensing volume, one or more IDU images corresponding to a current configuration of the three-dimensional control menu, the one or more IDU images providing indicia of any one or more selected menu items;

wherein the controller is configured to determine selection of any particular one of the menu items and to deliver a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices;

wherein the one or more medical equipment devices comprise a plurality of medical equipment devices; and

wherein the controller is configured to:

provide a first, top level, plurality of menu items, each first, top level, menu item selectable by the practitioner by one or more hand gestures made by the practitioner in a volumetric spatial region corresponding to the first, top level, menu item in the sensing volume and interpret selection of any particular one of the first, top level, menu items by delivering a corresponding operational command to select a corresponding one of the plurality of medical equipment devices; and

after selection of any particular one of the first, top level, menu items, provide a second, sub level, plurality of menu items, each second, sub level, menu item corresponding to a specific operational command of the corresponding one of the plurality of medical equipment devices and each second, sub level, menu item selectable by the practitioner by one or more hand gestures made by the practitioner in a volumetric spatial region corresponding to the second, sub level, menu item in the sensing volume, and interpret selection of any particular one of the second, sub level, menu items by delivering a corresponding specific operational command to the corresponding one of the plurality of medical equipment devices.

2. A system according to claim 1 wherein the one or more IDU images displayed outside of the sensing volume comprise, for at least one IDU image, a two-dimensional representation of the three-dimensional control menu.

3. A system according to claim 1 wherein the controller is configured, based on input from one or more sensors, to estimate at least one of a location of a head of the practitioner and an orientation of the head of the practitioner and to adjust the display of the one or more IDU images by the one or more IDU displays based at least in part on the at least one of the estimated location of the head of the practitioner and the estimated orientation of the head of the practitioner.

4. A system according to claim 3 wherein the controller is configured to adjust the display of the one or more IDU images based at least in part on the at least one of the estimated location of the head of the practitioner and the estimated orientation of the head of the practitioner by adjusting at least one of: a location, outside of the sensing volume, at which the one or more IDU images are displayed; and

an orientation of the one or more displayed IDU images.

5. A system according to claim 3 wherein the controller is configured to adjust the display of the one or more IDU images based at least in part on the at least one of the estimated location of the head of the practitioner and the estimated orientation of the head of the practitioner by adjusting one or more physical displays on which the one or more IDU images are displayed.

6. A system according to claim 3 wherein the one or more IDU displays are configured to display the one or more IDU images by projecting the IDU image onto a projection surface.

7. A system according to claim 6 wherein the controller is configured to adjust the one or more IDU images based at least in part on the at least one of the estimated location of the head of the practitioner and the orientation of the head of the practitioner by modifying the projection surface on which the one or more IDU images are projected.

8. A system according to claim 3 wherein the controller is configured to estimate the head orientation of the practitioner based on the estimated head location of the practitioner.

9. A system according to claim 1 wherein the controller is further configured, based on input from one or more sensors, to estimate a line of sight of a practitioner.

10. A system according to claim 1 comprising an augmented reality headset worn by the practitioner, wherein the one or more IDU displays are configured to display the one or more IDU images corresponding to the three-dimensional control menu by displaying one or more corresponding virtual IDU images to the practitioner by projecting the one or more IDU image from the augmented reality headset into one or both of the eyes of the practitioner.

11. A system according to claim 1 wherein a first IDU display is configured to display, in a first location outside of the sensing volume, a first device-specific IDU image by displaying the first device-specific IDU image on a first physical display outside of the sensing volume, the first physical display being at least one of: part of a first medical equipment device; part of a first control module connected to provide operational commands to the first medical equipment device.

12. A system according to claim 1 wherein the controller is configured to identify the practitioner from among one or more other humans in the operating room to be a controlling practitioner, wherein identifying the practitioner to be the controlling practitioner comprises at least one of: determining that the practitioner is the closest human to a reference location; identifying one or more fiducial markers associated with the practitioner; performing a facial recognition method on the practitioner; identifying one or more gestures performed by the practitioner; performing a voice recognition method on the practitioner; performing a body shape recognition method on the practitioner; and performing a retinal scan on the practitioner.

13. A system according to claim 1 wherein the at least one of the one or more medical equipment devices comprises a medical image display device for displaying medical images of a patient and wherein the corresponding operational command comprises a command which causes the medical image display device to perform one or more of: changing a medical image displayed on the medical image display device and changing characteristics of a medical image displayed on the medical image display device.

14. A system according to claim 1 wherein the one or more IDU images comprise medical image data corresponding to a patient.

15. A system for touchless control of one or more medical equipment devices in an operating room, the system comprising:

a 3D optical sensor connected for detecting one or more gestures made by a practitioner in a sensing volume of the sensor;

a controller connected to receive 3D optical data from the 3D optical sensor and configured to provide a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more hand gestures made by the practitioner in a volumetric spatial region corresponding to the menu item in the sensing volume; and

one or more interactive display unit (IDU) displays for displaying, outside of the sensing volume, one or more IDU images corresponding to a current configuration of the three-dimensional control menu, the one or more IDU images providing indicia of any one or more selected menu items;

wherein the controller is configured to determine selection of any particular one of the menu items and to deliver a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices;

wherein the controller is further configured, based on input from one or more sensors, to estimate a line of sight of a practitioner; and

wherein, when the estimated line of sight is directed within a first spatial range around a first medical equipment device, the controller is configured to determine that the practitioner is looking at the first medical equipment device and wherein, after determining that the practitioner is looking at the first medical equipment device, the controller is configured to provide a first device-specific three-dimensional control menu comprising first device-specific menu items which, when selected, result in delivering corresponding operational commands to the first medical equipment device to control operation of the first medical equipment device; and wherein, after determining that the practitioner is looking at the first medical equipment device, the controller is further configured to cause a first IDU display from among the one or more IDU displays to display, in a first location outside of the sensing volume, a first device-specific IDU image comprising graphics or text corresponding to the first device-specific menu items.

16. A system according to claim 15 wherein the controller is configured to estimate the line of sight of the practitioner by estimating a location and orientation of a head of the practitioner and estimating the line of sight of the practitioner to be along a line of sight vector, a start of the line of sight vector based on the estimated location of the head of the practitioner and an orientation of the line of sight vector based on the estimated orientation of the head of the practitioner.

17. A system according to claim 16 wherein the controller is configured to estimate the location and orientation of the head of the practitioner based at least in part on data obtained from at least one of: one or more 2D optical sensors; and one or more 3D optical sensors.

18. A system according to claim 15 where, when the estimated line of sight is directed within a second spatial range around a second medical equipment device which is different from the first medical equipment device, the controller is configured to determine that the practitioner is looking at the second medical equipment device and wherein, after determining that the practitioner is looking at the second medical equipment device, the controller is configured to:

provide a second device-specific three-dimensional control menu comprising second device-specific menu items different from the first device-specific menu items which, when selected, result in delivering corresponding operational commands to the second medical equipment device to control operation of the second medical equipment device; and

cause a second IDU display from among the one or more IDU displays to display, in a second location outside of the sensing volume, a second device-specific IDU image comprising graphics or text corresponding to the second device-specific menu items.

19. A system according to claim 15 wherein the at least one of the one or more medical equipment devices comprises a medical image display device for displaying medical images of a patient and wherein the corresponding operational command comprises a command which causes the medical image display device to perform one or more of: changing a medical image displayed on the medical image display device and changing characteristics of a medical image displayed on the medical image display device.

20. A system according to claim 15 wherein the one or more IDU images comprise medical image data corresponding to a patient.

21. A system for touchless control of one or more medical equipment devices in an operating room, the system comprising:

a 3D optical sensor connected for detecting one or more gestures made by a practitioner in a sensing volume of the sensor;

a controller connected to receive 3D optical data from the 3D optical sensor and configured to provide a three-dimensional control menu, the three-dimensional control menu comprising a plurality of menu items, each menu item selectable by the practitioner by one or more hand gestures made by the practitioner in a volumetric spatial region corresponding to the menu item in the sensing volume; and

one or more interactive display unit (IDU) displays for displaying, outside of the sensing volume, one or more IDU images corresponding to a current configuration of the three-dimensional control menu, the one or more IDU images providing indicia of any one or more selected menu items;

wherein the controller is configured to determine selection of any particular one of the menu items and to deliver a corresponding operational command to at least one of the one or more medical equipment devices to control operation of the at least one of the one or more medical equipment devices;

wherein the 3D optical sensor is mounted on a robotic positioning system for at least one of moving or orienting the 3D optical sensor, wherein the 3D optical sensor is configured to capture 3D optical data corresponding to a region of interest in the operation room and wherein the controller is configured to process the captured 3D optical data to locate and identify the one or more medical equipment devices in the operation room that are controllable using the three-dimensional control menu;

wherein the controller is configured to process the captured 3D optical data to locate and identify a particular medical equipment device that is controllable using the three-dimensional control menu;

wherein the controller is configured to provide the three-dimensional control menu by providing a particular device-specific three-dimensional control menu comprising particular device-specific menu items which, when selected, result in delivering corresponding operational commands to the particular medical equipment device to control operation of the particular medical equipment device; and

wherein the IDU display is configured to display the IDU image corresponding to the three-dimensional control menu by displaying a particular device-specific IDU image comprising graphics or text corresponding to the particular device-specific menu items.

22. A system according to claim 21 wherein the robotic positioning system comprises at least one of: an articulated robotic arm; and a series of linear actuators.

23. A system according to claim 21 wherein the robotic positioning system has two or more degrees of freedom.

24. A system according to claim 21 wherein the at least one of the one or more medical equipment devices comprises a medical image display device for displaying medical images of a patient and wherein the corresponding operational command comprises a command which causes the medical image display device to perform one or more of: changing a medical image displayed on the medical image display device and changing characteristics of a medical image displayed on the medical image display device.

25. A system according to claim 21 wherein the one or more IDU images comprise medical image data corresponding to a patient.