AUTONOMOUS PATIENT MONITORING WITH REMOTE CONSULTATION INTERFACE
Systems and methods for autonomous patient monitoring in a telehealth system. The system may include an in-room device including a camera and other devices to extract information from the patient environment in real-time. The information from the patient environment may be processed using object detection and other algorithms to identify and track relevant objects and/or events over time. The system may be configured to notify a care provider of specific events. A multi-modal large language model (MMLLM) may be employed to interpret the information about the patient environment and prompts or instructions from a care providers to monitor for and raise alerts or perform other actions in response to the detection of specific events. The system may also generate a user interface that enables a remote provider to quickly access zoom images of objects of interest within the patient environment.
1 . A telehealth device in a patient environment, the device comprising:
a camera that captures an image of the patient environment;
a controller in communication with the camera, the controller is configured to:
identify at least one object of interest in the image;
store a pan angle and a tilt angle associated with each of the at least one object of interest in association with the identity of each of the at least one object of interest;
transmit the image and the identity of each of the at least one object of interest to a remote device, wherein the remote device is configured to display a user interface that includes the image and an interactive icon for each of the at least one object of interest;
receive, from the remote device, a user selection of one of the interactive icons; and,
re-orient the camera based on the stored pan angle and tilt angle associated with the at least one object of interest associated with the selected interactive icon.
2 . The telehealth device of claim 1 , wherein identifying the at least one object of interest includes analyzing the image using a multimedia large language model (MMLLM).
3 . The telehealth device of claim 2 , wherein analyzing the image using the MMLLM includes prompting the MMLLM to identify objects of interest in a hospital environment.
4 . The telehealth device of claim 1 , wherein the processor is further configured to:
store a zoom factor in association with the identity of each of the at least one object of interest; and
adjust the camera zoom factor based on the stored zoom factor associated with the at least one object of interested associated with the selected interactive icon.
5 . The telehealth device of claim 1 , wherein identifying at least one object of interest includes determining a set of pixels representing a bounding box that encloses a portion of the image containing the object of interest.
6 . The telehealth device of claim 1 , wherein storing a pan angle and a tilt angle in association with each of the at least one object of interest includes storing a unique identifier in association with the pan angle and the tilt angle.
7 . A method for generating a user interface in a telehealth device, the method comprising:
capturing an image of a patient environment with a camera of a telehealth device;
identifying at least one object of interest in the image;
storing a pan angle and a tilt angle associated with each of the at least one object of interest in association with the identify of each of the at least one object of interest;
transmitting the image and the identity of each of the at least one object of interest to a remote device, wherein the remote device is configured to display a user interface that includes the image and an interactive icon for each of the at least one object of interest;
receiving, from the remote device, a user selection of one of the interactive icons; and,
re-orienting the camera based on the stored pan angle and tilt angle associated with the at least one object of interest associated with the selected interactive icon.
8 . The method of claim 7 , wherein identifying the at least one object of interest includes analyzing the image using a multimedia large language model (MMLLM).
9 . The method of claim 8 , wherein analyzing the image using the MMLLM includes prompting the MMLLM to identify objects of interest in a hospital environment.
10 . The method of claim 7 , wherein the processor is further configured to:
store a zoom factor in association with the identity of each of the at least one object of interest; and
adjust the camera zoom factor based on the stored zoom factor associated with the at least one object of interest associated with the selected interactive icon.
11 . The method of claim 7 , wherein identifying at least one object of interest includes determining a set of pixels representing a bounding box that encloses a potion of the image containing the object of interest.
12 . The method of claim 7 , wherein storing a pan angle and a tilt angle in association with each of the at least one object of interest includes storing a unique identifier in association with the pan angle and the tilt angle.