Gesture control of medical displays
The present invention relates to a computer-implemented medical method, a computer program and a system ( 1 ) for controlling a display ( 6 ) in a medical environment based a user's ( 8 ) viewing direction and gestures performed by the user ( 8 ), particularly involving the use of AR-goggles ( 9 ).
1. A computer-implemented method of controlling a display in a medical environment, the method comprising the following steps:
position data is acquired, describing a line of sight of an AR-device and/or a line of sight of a user's eyes with respect to the display;
displaying, via the AR-device when it is determined based on the position data that a user is looking at the display, a virtual control interface for controlling by gestures at least one of the display and content displayed on the display;
gesture data is acquired, describing at least one gesture performed by the user interacting with the virtual control interface displayed via the AR-device;
control data is determined based on the position data and the gesture data, describing a user's command to control at least one of the display and the displayed content.
2. The method according to claim 1 , wherein acquiring the position data involves determining
spatial position of the display and the spatial position of the AR-device in a common co-ordinate system, particularly by means of a medical tracking system;
the spatial position of the display in a co-ordinate system assigned to the AR-device.
3. The method according to claim 1 , wherein acquiring the position data and/or acquiring the gesture data involves an analysis of images, particularly a shape- or pattern-recognition-procedure in images acquired from at least one video-camera.
4. The method according to claim 1 , wherein the position data and/or the gesture data is acquired from at least one sensor of the AR-device, particularly wherein the AR-device comprises at least one of:
a video-camera, particularly oriented towards a field of view provided by the AR-device;
a video-camera oriented towards an eye of the user;
an acceleration sensor for determining spatial orientation of the AR-device;
a depth sensor for determining the distance of the AR-device to an object in the field of view provided by the AR-device.
5. The method according to claim 4 , wherein the content displayed by the display is supplemented by content displayed by the AR-device, particularly supplemented in a positionally registered manner within the field of view provided by the AR-device.
6. The method according to claim 4 , wherein the content displayed by the display is replaced by content displayed by the AR-device, particularly replaced in a positionally registered manner within the field of view provided by the AR-device.
7. The method according to claim 1 , wherein a plurality of displays is controlled.
8. The method according to claim 1 , wherein content displayed by a first display is transferred to a second display or, in a virtual manner, to another, invariant spatial position within a medical workspace defined by the user.
9. The method according to claim 7 , wherein at least during the transfer-time, the displayed content is displayed by the AR-device.
10. The method according to claim 1 , involving a plurality of AR-devices, wherein at least one of the plurality of AR-devices indicates where one or more users of the at least one other AR-devices is/are looking at.
11. The method of claim 1 further including providing the AR-device, particularly of AR-goggles, for controlling the display in the medical environment.
12. A non-transitory computer readable medium comprising instructions which when executed by at least one processor, cause the at least one processor to:
acquire position data, describing a line of sight of an AR-device and/or a line of sight of a user's eyes with respect to a display;
display, via the AR-device and if it is determined based on the position data that a user is looking at the display, a virtual control interface for controlling by gestures at least one of the display and content displayed on the display;
acquire gesture data describing at least one gesture performed by the user interacting with the virtual control interface displayed via the AR-device;
determine control data based on the position data and the gesture data, describing a user's command to control at least one of the display and the displayed content.
13. A medical system, comprising:
at least one computer having at least one processor and associated memory, the memory storing instructions which when executed by the at least one processor causes the at least one processor to:
acquire position data, describing a line of sight of an AR-device and/or a line of sight of a user's eyes with respect to a display;
display, via the AR-device and if it is determined based on the position data that a user is looking at the display, a virtual control interface for controlling by gestures at least one of the display and content displayed on the display;
acquire gesture data describing at least one gesture performed by the user interacting with the virtual control interface displayed via the AR-device;
determine control data based on the position data and the gesture data, describing a user's command to control at least one of the display and the displayed content;
at least one display for displaying medical content in a medical environment; and
at least one AR-device comprising a tracking system for controlling the display,
wherein the at least one computer is operably coupled to
a medical tracking system for acquiring, from the medical tracking system, at least the position data and the gesture data, and
the display for issuing a control signal to the display for controlling the operation of the display on the basis of the position data and the gesture data.