Augmented reality system for displaying patient data
System and methods are provided for augmented reality displays for medical and physiological monitoring. Augmented reality user interfaces are virtually pinned to a physical device, a location, or to a patient. An augmented reality position determination process determines the presentation of user interfaces relative to reference positions and reference objects. Detection of gestures causes the augmented reality users interfaces to be updated, such as pinning a user interface to a device, location, or patient. Looking away from an augmented reality user interface causes the user interface to minimize or disappear in an augmented reality display. An augmented reality gesture detection process determines gestures based on captured image data and computer vision techniques performed on the image data.
1. A method for presenting augmented reality data from a medical monitoring device, the method comprising:
under control of a hardware processor,
receiving physiological monitoring data comprising physiological parameter values associated with a patient from a monitoring hub;
receiving hub user interaction data from the monitoring hub, wherein the hub user interaction data comprises a plurality of user interface elements in a first arrangement;
determining a hub user interface configuration from the hub user interaction data;
causing presentation of a user interface on a display of the monitoring hub according to the hub user interface configuration;
accessing user interface configuration data, wherein the user interface configuration data is indicative of the hub user interface configuration;
generating, from the physiological monitoring data, a plurality of augmented reality objects according to the user interface configuration data, wherein each object from the plurality of augmented reality objects corresponds to an element from the plurality of user interface elements;
receiving user interaction data from a user input device of an augmented reality device, wherein the user interaction data comprises an indication of an interaction to virtually pin the plurality of augmented reality objects to a reference object;
determining a reference position based at least in part the user interaction data; and
causing presentation of the plurality of augmented reality objects in an augmented reality display, wherein the plurality of augmented reality objects are presented relative to the reference position in a second arrangement corresponding to the first arrangement.
2. The method of claim 1 , further comprising:
identifying, from the user interaction data, the reference object;
determining the reference position for the reference object; and
calculating a positional offset from the reference position, wherein the plurality of augmented reality objects are presented relative to the reference position according to the positional offset.
3. The method of claim 1 , further comprising:
determining, from the user interface configuration data, whether to present a direct overlay;
identifying, from the user interface configuration data, the reference object;
calculating the reference position for the reference object, wherein an overlay object of the plurality of augmented reality objects is presented at the reference position.
4. The method of claim 1 , wherein the reference object is the patient.
5. The method of claim 1 , further comprising:
detecting a gesture from the user interaction data; and
generating the user interface configuration data based at least in part on the gesture.
6. The method of claim 5 , wherein the user input device comprises a camera.
7. The method of claim 6 , wherein the user input data comprises image data, and wherein detecting the gesture from the user input data further comprises:
determining color histogram data from the image data;
locating a search window in the image data according to the color histogram data; and
identifying a plurality of positions of the search window in the image data.
8. The method of claim 1 , further comprising:
receiving second user interaction data from the user input device, the second user interaction data indicating a third arrangement of the plurality of augmented reality objects;
generating second user interface configuration data from the second user interaction data; and
causing presentation, in the augmented reality display, of the plurality of augmented reality objects in the third arrangement according to the second user interface configuration data.
9. The method of claim 1 , wherein each element of the plurality of user interface elements comprises a physiological parameter value.
10. The method of claim 1 , wherein the hub user interaction data is indicative of at least one addition, removal, or rearrangement of a user interface element.
11. A system configured to present physiological data, the system comprising:
a memory device configured to store instructions; and
a hardware processor configured to execute the instructions to:
receive physiological monitoring data comprising physiological parameter values associated with a patient from a monitoring hub;
receive hub user interaction data from the monitoring hub, wherein the hub user interaction data comprises a plurality of user interface elements in a first arrangement;
determine a hub user interface configuration from the hub user interaction data;
cause presentation of a user interface on a display of the monitoring hub according to the hub user interface configuration;
access user interface configuration data, wherein the user interface configuration data is indicative of the hub user interface configuration;
generate, from the physiological monitoring data, a plurality of augmented reality objects according to the user interface configuration data, wherein each object from the plurality of augmented reality objects corresponds to an element from the plurality of user interface elements;
receive user interaction data from a user input device of an augmented reality device, wherein the user interaction data comprises an indication of an interaction to virtually pin the plurality of augmented reality objects to a reference object;
determining a reference position based at least in part the user interaction data; and
cause presentation of the plurality of augmented reality objects in an augmented reality display, wherein the plurality of augmented reality objects are presented relative to the reference position in a second arrangement corresponding to the first arrangement.
12. The augmented reality device of claim 11 , wherein the hardware processor is further configured to:
generate at least part of the user interface configuration data from the user interaction data.
13. The augmented reality device of claim 11 , wherein the hardware processor is further configured to:
calculate a positional offset from the reference position, wherein the plurality of augmented reality objects are presented relative to the reference position according to the positional offset.
14. The augmented reality device of claim 11 , wherein the hardware processor is further configured to:
determine, from the user interface configuration data, whether to present a direct overlay, wherein an overlay object of the plurality of augmented reality objects is presented at the reference position.
15. The augmented reality device of claim 11 , wherein the hardware processor is further configured to:
receive user input data from a user input device;
detect a gesture from the user input data; and
generate the user interface configuration data based at least in part on the gesture.
16. The augmented reality device of claim 15 , wherein the user input device comprises a camera.
17. The augmented reality device of claim 15 , wherein the user input data comprises image data, and wherein detecting the gesture from the user input data further comprises:
determining color histogram data from the image data;
locating a search window in the image data according to the color histogram data; and
identifying a plurality of positions of the search window in the image data.
18. A system for presenting augmented reality data, the system comprising:
a medical monitoring device comprising a first hardware processor and a first display, the first hardware processor of the medical monitoring device configured to:
receive monitor user interaction data;
determine a monitor user interface configuration from the monitor user interaction data, wherein the monitor user interaction comprises a plurality of user interface elements in a first arrangement; and
cause presentation of a user interface on the first display according to the monitor user interface configuration; and
an augmented reality device comprising a second hardware processor, memory, a second display, and a wireless device configured to communicate with the medical monitoring device in communication with a patient, the second hardware processor of the augmented reality device configured to:
identify a user interaction with the augmented reality device by a clinician wearing the augmented reality device, the user interaction comprising a gesture;
determine that the user interaction is an instruction to virtually pin the user interface of the medical monitoring device to a virtual location separate from the medical monitoring device;
wirelessly obtain patient data depicted in the user interface from the medical monitoring device;
generate, from the patient data, a plurality of augmented reality objects according to the monitor user interface configuration, wherein each object from the plurality of augmented reality objects corresponds to an element from the plurality of user interface elements; and
output for presentation to the clinician, in the second display of the augmented reality device, the plurality of augmented reality objects pinned to the virtual location, wherein the plurality of augmented reality objects are presented in a second arrangement corresponding to the first arrangement.
19. The system of claim 18 , wherein the instruction is to virtually pin the user interface outside of and next to a hospital room of the patient so that the clinician can subsequently view the user interface while walking past the hospital room and without entering the hospital room.
20. The system of claim 18 , wherein the plurality of augmented reality objects pinned to the virtual location is viewable only when a field of view of the second display includes the virtual location, and wherein the virtual location does not obscure the patient.
21. The system of claim 18 , wherein the augmented reality device is further operable to detect that the clinician has moved his or her head away from looking at the medical monitoring device and to reduce a size or content of the plurality of augmented reality objects at the virtual location.
22. The system of claim 21 , wherein the augmented reality device further comprises a camera or a movement sensor configured to detect that the clinician has moved his or her head away from looking at the medical monitoring device.